System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting audio-data of a user; means for transmitting the collected audio-data to a server; means for analyzing the audio-data using a AI model generated at the server and suggesting optimal hearing device settings; and means for automatically adjusting hearing device settings based on the analysis result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern hearing aids lack the functionality to automatically optimize settings to suit the user's hearing environment and individual needs. Furthermore, the risk of falls is increasing, particularly among the elderly, due to hearing loss and impaired balance, creating a need for features to prevent falls. Furthermore, hearing aid users have difficulty properly understanding conversations in everyday life and business situations, so they need assistance functions to effectively grasp the content of conversations and meetings. To address these issues, we offer an innovative hearing aid system that utilizes generative AI models. [Means for solving the problem]

[0005] The present invention includes a means for collecting a user's hearing data and transmitting it to a server. It also includes a means for analyzing the hearing data using an AI model generated by the server based on that data and proposing optimal hearing aid settings. It also provides a means for automatically adjusting the hearing aid settings based on the analysis results. This allows users to obtain optimal hearing aid settings without the need for expert support.

[0006] The device also has a means for collecting user movement data in real time and sending it to a server, and a means for assessing the risk of falling using an AI model generated by the server. It also has a means for alerting the user when a high risk of falling is detected, and a means for sending an alert to emergency contacts when a fall occurs. This ensures the user's safety and prevents accidents caused by falls.

[0007] Furthermore, it includes a means for collecting conversational audio in real time and a means for converting the audio data into text data. It transmits the text data to a server, analyzes the text data using an AI model generated on the server, and extracts key points. It also includes a means for converting the content into easy-to-understand illustrations, and provides a means for displaying the converted illustrations and transcription data to the user. This allows users to easily understand the content of conversations and meetings.

[0008] A "server" is a computer device that receives data sent from a hearing aid terminal, analyzes and evaluates the data using a generative AI model, and returns the results to the hearing aid terminal.

[0009] A "terminal" is a device that includes a hearing aid worn by a user, a smartphone that connects to the hearing aid, a PC, or other computer device.

[0010] "User" refers to an individual who uses the hearing aid system of the present invention, and includes primarily, but is not limited to, people with hearing loss and the elderly.

[0011] "Auditory data" refers to data related to the sounds heard by the user, such as the sound environment around the user, volume, and frequency spectrum.

[0012] "Motion data" is data collected by monitoring a user's movements with sensors or cameras, and includes information on the user's body movements and posture.

[0013] A "generative AI model" refers to an artificial intelligence algorithm that analyzes collected data and performs tasks such as optimizing hearing aid settings, assessing fall risk, and analyzing conversation content.

[0014] "Optimal hearing aid settings" are those settings that allow the hearing aid to function most effectively based on the user's individual hearing environment and volume preferences.

[0015] "Fall risk" is an evaluation value that indicates the likelihood that the user will fall, and is evaluated by a generative AI model based on movement data.

[0016] The "means of warning" is a function that notifies the user of a high risk of falling using sound, vibration, etc.

[0017] "Means for sending alerts to emergency contacts" is a function that automatically notifies registered contacts via text message or phone call when the user falls.

[0018] "Audio data" refers to the user's conversational voice or the voice of an online conference, and is audio information collected by the hearing aid terminal.

[0019] "Text data" refers to data resulting from the transcription of collected audio data.

[0020] "Key points" refer to particularly meaningful information in the transcribed text data and key content that users should understand.

[0021] "Means for converting into illustrations" is a function that converts important points extracted from text data into illustrations that are easy to understand visually.

[0022] "Display means" refers to a function that displays text data and illustrations on the screen of a hearing aid terminal or linked device. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] The present invention is an innovative hearing aid system that utilizes generative AI models and is equipped with multiple functions. Specific embodiments are described below.

[0045] Optimizing hearing aid settings

[0046] This hearing aid system has the ability to automatically optimize hearing aid settings based on the user's hearing environment by simply putting on the hearing aid and launching the settings app. When the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal hearing aid settings. The analysis results are sent back to the device, which automatically adjusts the hearing aid settings. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[0047] For example, if a user finds it difficult to hear people during a meeting, they can optimize their hearing aid settings. The hearing aid device then analyzes the sound environment of the meeting, and the AI ​​model proposes optimal volume and filter settings. The hearing aid device then applies the proposed settings, and the user can confirm the actual effect.

[0048] Fall risk detection and prevention

[0049] The system detects and prevents the risk of falls by collecting user movement data in real time and sending it to a server. Cameras and sensors built into the hearing aid device monitor the user's movements and collect movement data. This data is temporarily stored in the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends an alert to the device, which then alerts the user via audio and vibration. Additionally, if the user actually falls, the hearing aid device immediately sends an alert to emergency contacts.

[0050] For example, if a user is about to lose their balance while descending stairs, the hearing aid device will immediately sound an alarm to alert the user, and if they still fall, an alert will be sent to their family or caregiver.

[0051] Transcription and illustration of conversation content

[0052] The system collects conversational audio in real time, transcribes it, and converts it into illustrations to help users understand the content of the conversation. When a user puts on their hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[0053] For example, if a user is having difficulty understanding the conversation while dining with a friend at a restaurant, the hearing aid will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will appear on the screen.

[0054] Summarizing and illustrating online meetings

[0055] The system collects audio from online meetings in real time, converts the content into text, summarizes it, and displays it as illustrations. Users participate in online meetings, and their hearing aids connect to their PCs or smartphones to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen by the hearing aid terminal or a connected device.

[0056] For example, when a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed, along with an illustration of the new product.

[0057] The above is a detailed description of the mode for carrying out the invention.

[0058] The processing flow will be explained below.

[0059] Optimizing hearing aid settings

[0060] Processing Steps

[0061] Step 1:

[0062] The user puts on the hearing aid and launches the settings app on their smartphone or the hearing aid.

[0063] Step 2:

[0064] The user presses the "Optimize Settings" button in the app.

[0065] Step 3:

[0066] The device (hearing aid or smartphone) records the surrounding sound environment for a few seconds and collects auditory data such as volume and frequency spectrum.

[0067] Step 4:

[0068] The terminal transmits the collected auditory data to a server.

[0069] Step 5:

[0070] Based on the data received by the server, a generative AI model is run, which analyzes the auditory data and suggests optimal settings.

[0071] Step 6:

[0072] The server returns the configuration data including the analysis results to the terminal.

[0073] Step 7:

[0074] The device applies the received configuration data and automatically adjusts the audio settings of the hearing aid.

[0075] Step 8:

[0076] The user will see a "Setup complete" notification and be asked to review their listening progress and provide feedback.

[0077] Step 9:

[0078] If the user instructs fine adjustment as needed, the setting data is sent to the server again and adjustments are made using the same process.

[0079] Fall risk detection and prevention

[0080] Processing Steps

[0081] Step 1:

[0082] The user wears the hearing aid and goes about their daily activities.

[0083] Step 2:

[0084] The device (camera and sensors built into the hearing aid) collects the user's behavior data in real time.

[0085] Step 3:

[0086] The operating data collected by the device is temporarily stored and an initial analysis is performed using an anomaly detection algorithm.

[0087] Step 4:

[0088] The device sends the analysis results to the server and requests a detailed analysis.

[0089] Step 5:

[0090] Based on the data received by the server, a generative AI model is run to assess the risk of falling.

[0091] Step 6:

[0092] The server returns the results of the risk assessment to the terminal.

[0093] Step 7:

[0094] If the device is judged to be at high risk of falling, it will issue an alert to the user via voice or vibration to warn them.

[0095] Step 8:

[0096] If the user falls, the device will detect the fall and automatically send a text message or phone alert to pre-defined emergency contacts.

[0097] Transcription and illustration of conversation content

[0098] Processing Steps

[0099] Step 1:

[0100] The user puts on the hearing aid and begins a conversation.

[0101] Step 2:

[0102] The device (hearing aid) collects conversational audio in real time.

[0103] Step 3:

[0104] The device temporarily stores the voice data and performs initial voice recognition.

[0105] Step 4:

[0106] The terminal transmits the converted text data to the server.

[0107] Step 5:

[0108] Based on the text data received by the server, a generative AI model is run to analyze keywords and important points.

[0109] Step 6:

[0110] Based on the analysis results, the server converts the content into easy-to-understand illustrations.

[0111] Step 7:

[0112] The server sends the generated illustrations and transcription data back to the device.

[0113] Step 8:

[0114] The device (hearing aid or connected smartphone) displays the transcription results and illustrations on the user's screen.

[0115] Summarizing and illustrating online meetings

[0116] Processing Steps

[0117] Step 1:

[0118] A user joins an online meeting.

[0119] Step 2:

[0120] The user connects the conferencing app on their PC or smartphone to their hearing aid.

[0121] Step 3:

[0122] The device (hearing aid or PC / smartphone) collects the conference audio in real time.

[0123] Step 4:

[0124] The device transcribes the audio data and sends the generated text data to the server.

[0125] Step 5:

[0126] Based on the text data received by the server, a generative AI model is run to generate a summary of the meeting contents.

[0127] Step 6:

[0128] Based on the summary results, the server illustrates important points and keywords.

[0129] Step 7:

[0130] The server sends the summary and illustrations back to the device.

[0131] Step 8:

[0132] The device (hearing aid or connected smartphone) displays a summary and illustrations on the user's screen, allowing the user to efficiently understand the meeting content.

[0133] Example 1

[0134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] Conventional hearing aid systems have difficulty in properly adjusting to the user's individual hearing environment and operating conditions. Furthermore, proper settings according to the environment are often performed manually, placing a burden on the user. Furthermore, functions for detecting the risk of falls and assisting in understanding conversation content are lacking, making them insufficient for improving the user's safety and quality of daily life. A comprehensive system that solves these problems is needed.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0137] In this invention, the server includes means for analyzing the user's hearing data and proposing optimal hearing aid settings, means for analyzing the user's motion data and assessing the risk of falling, means for analyzing conversational audio, extracting key points and illustrating them, and means for analyzing conference audio, creating summaries and illustrating them, thereby improving the user's hearing environment and safety, and enabling them to understand the content of conversations and conferences.

[0138] "Hearing data" refers to data related to the user's hearing, such as volume and frequency spectrum obtained through a hearing aid.

[0139] "Motion data" is data that includes information about the user's body movements and posture.

[0140] A "server" is a computer system that works in conjunction with hearing aid terminals via a network to analyze and process data.

[0141] A "generative AI model" is a machine learning algorithm that analyzes collected data and generates optimal suggestions and results.

[0142] "Analysis" is the process of investigating and examining collected data in detail using specialized methods to derive useful information and conclusions.

[0143] "Hearing aid settings" are settings that adjust various parameters of the hearing aid (e.g., volume, frequency filters, etc.).

[0144] "Fall risk" is an assessment that indicates the degree of possibility that the user will fall.

[0145] An "alert" is a notification that warns or alerts the user.

[0146] "Conversational voice" refers to the voice that a user utters while communicating with others.

[0147] "Text data" refers to data obtained by converting voice data into text format.

[0148] "Important points" are parts of a conversation or meeting that are judged to be particularly important.

[0149] An "illustration" is an image that visually represents an important point or summary.

[0150] A "summary" is a concise summary of long information such as that from a meeting.

[0151] The present invention is an innovative hearing assistance system that utilizes a generative AI model and is equipped with multiple functions. Specific embodiments are described below.

[0152] Optimizing hearing aid settings

[0153] This hearing assistance system has the function of automatically optimizing settings based on the hearing environment when the user wears the hearing assistance device and launches the settings app on their smartphone. Specifically, when the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal settings. The analysis results are sent back to the device, which automatically adjusts the settings of the hearing assistance device. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[0154] For example, if a user finds it difficult to hear people during a meeting, they can optimize their settings. The device will then analyze the sound environment of the meeting, and the AI ​​model will suggest optimal volume and filter settings. The device will then apply the suggested settings, and the user can check the actual effect.

[0155] Fall risk detection and prevention

[0156] The system has the ability to detect and prevent the risk of falls by collecting user movement data in real time and sending it to a server. The device's built-in cameras and sensors monitor the user's movements and collect movement data. This data is temporarily stored on the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends a warning alert to the device, and the device alerts the user via audio and vibration. Additionally, if the user actually falls, the device immediately sends an alert to emergency contacts.

[0157] For example, if a user is about to lose balance while descending stairs, the device will immediately sound an alarm to warn the user, and if the user still falls, an alert will be sent to a family member or caregiver.

[0158] Transcription and illustration of conversation content

[0159] The system collects conversational audio in real time, transcribes it, and converts it into easy-to-understand illustrations. When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[0160] For example, if a user is having difficulty understanding the conversation while dining with friends at a restaurant, the hearing aid device will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will be displayed on the screen.

[0161] Summarizing and illustrating online meetings

[0162] The system has the ability to collect audio from online meetings in real time, convert the content into text, summarize it, and display it as illustrations. Users participate in online meetings, and the hearing aid connects with a PC or smartphone to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen via their terminal or a connected device.

[0163] For example, when a user participates in a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed on the screen, along with an illustration of the new product.

[0164] Prompt Sentence Examples

[0165] "Please suggest the best assistive hearing device settings for me during the meeting."

[0166] "Send an alert when the user loses balance while descending stairs."

[0167] "Please transcribe the conversation in the restaurant and show it in illustrations."

[0168] "Please summarize and illustrate the contents of the Zoom meeting."

[0169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0170] Optimizing hearing aid settings

[0171] Step 1:

[0172] The user launches the Settings app on their smartphone and presses the "Optimize Settings" button.

[0173] Input: User actions

[0174] Output: Triggers configuration optimization

[0175] At this point, the setting optimization process is initiated by the user.

[0176] Step 2:

[0177] The device will use the built-in microphone to record the surrounding sound environment for a few seconds.

[0178] Input: Trigger settings optimization

[0179] Output: Sound environment data

[0180] Specifically, the device captures ambient sounds during the meeting.

[0181] Step 3:

[0182] The device extracts auditory data such as volume and frequency spectrum.

[0183] Input: Sound environment data

[0184] Output: Auditory data

[0185] The collected sound environment data is analyzed and the necessary auditory data is extracted.

[0186] Step 4:

[0187] The terminal compresses the auditory data and prepares it for transmission to the server.

[0188] Input: Auditory data

[0189] Output: Compressed data

[0190] Data such as volume and frequency spectrum is compressed and converted into a format that is sent to the server.

[0191] Step 5:

[0192] The terminal transmits the compressed data to the server.

[0193] Input: Compressed data

[0194] Output: Data on the server

[0195] Data is sent from the terminal to the server.

[0196] Step 6:

[0197] The server unpacks the received data and begins analyzing it.

[0198] Input: Compressed data

[0199] Output: Expanded data

[0200] Extract the data and prepare it for analysis.

[0201] Step 7:

[0202] An AI model generated on the server analyzes the deployment data and calculates the optimal settings.

[0203] Input: Expanded data

[0204] Output: Optimal Settings

[0205] An AI model examines the data and determines the optimal hearing aid settings for the user.

[0206] Step 8:

[0207] The server returns the analysis results to the device.

[0208] Input: Optimal Settings

[0209] Output: Configuration data on the terminal

[0210] The server communicates the optimal settings to the device.

[0211] Step 9:

[0212] The settings of the hearing aid device are automatically adjusted based on the analysis results received by the device.

[0213] Input: Setting data

[0214] Output: Calibrated hearing aids

[0215] Your device automatically changes the settings of your hearing aid.

[0216] Step 10:

[0217] The user will be notified that the setup is complete.

[0218] Input: Adjusted hearing aid

[0219] Output: Notification

[0220] The user is notified when the settings are complete, and can make fine adjustments as needed.

[0221] Fall risk detection and prevention

[0222] Step 1:

[0223] Cameras and sensors built into the device monitor and collect information about the user's behavior.

[0224] Input: User action

[0225] Output: Operation data

[0226] Obtain user behavior data in real time.

[0227] Step 2:

[0228] The operating data collected by the device is temporarily saved and an initial analysis is performed.

[0229] Input: Operation data

[0230] Output: Initial analysis results

[0231] An initial analysis is performed using an anomaly detection algorithm on the device.

[0232] Step 3:

[0233] The device sends the initial analysis results to the server.

[0234] Input: Initial analysis results

[0235] Output: Data on the server

[0236] The initial analysis results are sent to the server to prepare for detailed analysis.

[0237] Step 4:

[0238] The server expands the received data and performs a detailed fall risk assessment using a generative AI model.

[0239] Input: Initial analysis results

[0240] Output: Detailed risk assessment

[0241] The server performs detailed analysis and assesses the risk of falling.

[0242] Step 5:

[0243] If a high risk of falling is detected, the server generates an alert to warn the user.

[0244] Input: Detailed risk assessment

[0245] Output: Alert

[0246] The server generates an alert and sends it to the device.

[0247] Step 6:

[0248] The device receives an alert and sends it to the user via sound or vibration.

[0249] Input: Alert

[0250] Output: Caution

[0251] The device will notify the user of the alert by sound or vibration.

[0252] Step 7:

[0253] If the user actually falls, the device will automatically make an emergency call.

[0254] Input: Fall detection

[0255] Output: Emergency alert

[0256] It detects if the user falls and sends an alert to emergency contacts.

[0257] Transcription and illustration of conversation content

[0258] Step 1:

[0259] A user wears a hearing aid and the terminal collects conversational audio.

[0260] Input: Conversational audio

[0261] Output: Audio data

[0262] To acquire a user's conversation voice in real time.

[0263] Step 2:

[0264] The device converts the collected voice data into text.

[0265] Input: Audio data

[0266] Output: Text data

[0267] Voice recognition software is used to convert voice data into text data.

[0268] Step 3:

[0269] Sends text data to the server.

[0270] Input: Text data

[0271] Output: Data on the server

[0272] The text data is sent to the server.

[0273] Step 4:

[0274] An AI model generated on the server analyzes the text data and extracts key points.

[0275] Input: Text data

[0276] Output: Key Points

[0277] An AI model is used to analyze text data and extract key points from the conversation.

[0278] Step 5:

[0279] The server converts the extracted key points into illustrations.

[0280] Input: Key Points

[0281] Output: illustration data

[0282] Convert key points into illustrations to make them easier to understand visually.

[0283] Step 6:

[0284] The device displays the converted illustration and transcription data on the user's screen.

[0285] Input: illustration data, transcription data

[0286] Output: What the user sees on their screen

[0287] The device displays the illustration and transcription results to provide the user.

[0288] Summarizing and illustrating online meetings

[0289] Step 1:

[0290] A user participates in an online conference, and a terminal collects the conference audio.

[0291] Input: Conference audio

[0292] Output: Audio data

[0293] Collect the user's online conference audio.

[0294] Step 2:

[0295] The terminal converts the collected voice data into text data.

[0296] Input: Audio data

[0297] Output: Text data

[0298] Use speech recognition software to convert meeting audio into text data.

[0299] Step 3:

[0300] Sends text data to the server.

[0301] Input: Text data

[0302] Output: Data on the server

[0303] The text data is sent to the server.

[0304] Step 4:

[0305] The AI ​​model generated on the server analyzes the text data and creates a summary.

[0306] Input: Text data

[0307] Output: Summary data

[0308] The AI ​​model analyzes the meeting content and generates a summary.

[0309] Step 5:

[0310] The server converts the summarized content into illustrations.

[0311] Input: Summary data

[0312] Output: illustration data

[0313] Convert summarized meeting content into visually easy-to-understand illustrations.

[0314] Step 6:

[0315] The terminal displays the converted illustration and summary data on the user's screen.

[0316] Input: illustration data, summary data

[0317] Output: What the user sees on their screen

[0318] The terminal displays the illustration and summary content for the user.

[0319] (Application example 1)

[0320] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] Conventional hearing aid systems are limited to analyzing voice and motion data and lack optimal functionality for real-world work environments and safety. Noisy factory environments, in particular, can make it difficult to hear voice instructions and increase the risk of workers falling. A method is needed to resolve these issues and improve work efficiency and safety.

[0322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0323] In this invention, the server

[0324] A means to analyze environmental sounds in the factory in real time and provide voice instructions to workers with optimal volume and sound quality,

[0325] A method for collecting worker movement data in real time using sensors and assessing the risk of falls.

[0326] A means to transcribe voice instructions and meeting content in real time and convert them into easy-to-understand illustrations for display,

[0327] This enables optimization of voice instructions according to the work environment, improvement of worker safety, and visualization of information.

[0328] "Auditory data" is acoustic information obtained through the user's ears, and is data that includes characteristics such as volume and frequency spectrum.

[0329] A "server" is a computer system on a network that collects, analyzes, and stores data.

[0330] An "AI model" is a model that uses machine learning algorithms to analyze data and derive optimal solutions to specific problems.

[0331] "Hearing aid settings" are various adjustment items for assisting the user's hearing, such as the volume and frequency filter settings of the hearing aid.

[0332] "Motion data" is data that includes information about the user's body movements and posture.

[0333] The "fall risk" is an evaluation value of the possibility that the user will fall, and is calculated based on the motion data.

[0334] A "warning" is an alert or notification issued to alert the user.

[0335] "Transcription" is the process of converting audio into text data.

[0336] An "illustration" is a diagram or picture that visually represents information.

[0337] A "sensor" is a device that senses physical data (e.g., motion or sound) and converts that data into an electronic signal.

[0338] "Analysis" is the process of examining collected data in detail to understand its meaning and patterns.

[0339] "Voice instructions" are instructions or commands given to the user using voice.

[0340] The system for realizing the present invention is configured using the following hardware and software.

[0341] Hardware

[0342] 1. Smartphone (e.g. iPhone or Android device)

[0343] 2. Smart glasses (e.g. Microsoft HoloLens)

[0344] 3. Sensors (accelerometer, gyro sensor, camera, etc.)

[0345] software

[0346] 1. Speech recognition library (e.g. Google Speech-to-Text API)

[0347] 2. Machine learning frameworks (e.g., TensorFlow)

[0348] 3. Real-time motion analysis library (e.g., OpenCV)

[0349] System configuration and processing

[0350] 1. Collecting user hearing and movement data:

[0351] The user wears the hearing aid, and the system collects real-time hearing and motion data through motion sensors and microphones, for example, continuously monitoring the worker's movements and surrounding sounds while working in a factory.

[0352] 2. Data transmission and analysis on the server:

[0353] The collected data is sent via smartphones or smart glasses to a server, which receives the collected data and analyzes it using a generative AI model (e.g., GPT-4).

[0354] 3. Recommend and apply optimal settings:

[0355] The server analyzes the data and then suggests optimal volume and filter settings. This makes it easier to hear work instructions, even in noisy factory environments. The analysis results are sent to the hearing aid device, which automatically adjusts the settings.

[0356] 4. Fall risk assessment and alert notification:

[0357] The system evaluates the risk of falls based on the worker's movement data, and if a high risk is detected, a warning is issued via smartphone or smart glasses. If a fall actually occurs, a notification is sent immediately to the manager.

[0358] 5. Transcription and visualization of audio instructions:

[0359] Work instructions and meeting content are transcribed in real time, and a generative AI model extracts key points and converts them into easy-to-understand illustrations. The converted illustrations and transcription data are displayed on the user's smartphone or smart glasses.

[0360] Specific examples

[0361] When workers performing maintenance on equipment in a factory have difficulty hearing instructions due to excessive noise, the system optimizes voice instructions and continuously monitors the risk of falls to enhance safety. Meetings are also transcribed in real time, and a generative AI model summarizes and displays relevant illustrations to enhance worker understanding.

[0362] Prompt Sentence Examples

[0363] Please convert the following audio data into text and summarize it. When doing so, please extract key points so that the summary can be converted into an easy-to-understand illustration.

[0364] Audio data: {Audio data}

[0365] This configuration makes it possible to optimize voice instructions to suit the work environment, improve worker safety, and visualize information, all of which are the goals of the invention.

[0366] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0367] Step 1:

[0368] The user puts on the hearing aid and launches the application installed on the device. The device begins collecting hearing data and movement data. Specifically, it uses the microphone and sensors built into the hearing aid to monitor the user's surrounding sound environment and movement in real time. Input: Environmental sound, movement data. Output: Collected raw data.

[0369] Step 2:

[0370] The device sends the collected auditory and movement data to a server. The data is transmitted over a network using a secure communication protocol (e.g., HTTPS). Input: Collected raw data. Output: Data transmitted to the server.

[0371] Step 3:

[0372] The server processes the received data and analyzes it using a generative AI model. It analyzes the data in real time using a speech recognition library (e.g., Google Speech-to-Text API) and a machine learning framework (e.g., TensorFlow). Input: Transmitted data. Output: Analysis results (optimal hearing aid settings, fall risk assessment, transcription results, etc.).

[0373] Step 4:

[0374] The server then proposes optimal hearing aid settings based on the analysis results. Specifically, the generative AI model calculates volume and filter settings appropriate for the sound environment and sends the settings information to the device. Input: Analysis results. Output: Hearing aid setting information.

[0375] Step 5:

[0376] The device uses the received hearing aid setting information to automatically adjust the hearing aid settings. The processor inside the hearing aid receives this and adjusts the volume and filter settings in real time. Input: Hearing aid setting information. Output: Automatically adjusted hearing aid settings.

[0377] Step 6:

[0378] The device also continues to analyze movement data and issues an alert to the user if it determines there is a high risk of falling. Specifically, it notifies the user by sound warning or vibration. In addition, if a fall actually occurs, it immediately sends a notification to emergency contacts. Input: Movement data analyzed in real time. Output: Warning alert, emergency notification.

[0379] Step 7:

[0380] When a user is participating in a conversation or meeting, the device converts the collected voice data into text and sends the text data to the server. Input: Voice data. Output: Text data.

[0381] Step 8:

[0382] The server receives text data and uses a generative AI model to extract key points and convert them into easy-to-understand illustrations. Input: Text data. Output: Key points, illustrations.

[0383] Step 9:

[0384] The device visually displays the illustrations and transcription data received from the server to the user. This information is displayed on the display of a smartphone or smart glasses to help the user understand. Input: Illustrations, transcription data. Output: Visually displayed information.

[0385] Through these processing steps, the system supports the user's hearing and movements in real time, providing optimal hearing aid settings, improved safety, and information visualization.

[0386] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0387] The present invention is an innovative hearing aid system that utilizes generative AI models and incorporates user emotion recognition functionality. Specific embodiments of each function are described in detail below.

[0388] Optimizing hearing aid settings

[0389] When a user puts on their hearing aid and launches the settings app, the device automatically optimizes the hearing aid settings based on their hearing environment. When the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated on the server analyzes the data and suggests optimal hearing aid settings. The analysis results are sent back to the device, which automatically adjusts the hearing aid settings.

[0390] For example, if a user points out that they have difficulty hearing voices in a movie theater, the hearing aid device analyzes the sound environment and the AI ​​model suggests optimal volume and filter settings. The hearing aid device then applies the settings and the user confirms the sound quality.

[0391] Fall risk detection and prevention

[0392] This includes a function to detect and prevent the risk of falls by collecting user movement data in real time and sending it to a server. Cameras and sensors built into the hearing aid device monitor the user's movements and collect movement data. This data is sent to the server, where it is analyzed by a generative AI model and used to assess the risk of falls. If a high risk of falling is detected, the server sends an alert to the device, which then issues an audio or vibration alert to the user. Additionally, if the user actually falls, the hearing aid device automatically sends an alert to emergency contacts.

[0393] For example, if a user is about to lose balance while descending stairs, the hearing aid device will immediately sound an alarm to alert the user, and if they still fall, an alert will be sent to emergency contacts.

[0394] Transcription and illustration of conversation content

[0395] The system collects conversational audio in real time, transcribes it, and converts it into illustrations to help users understand the content of the conversation. When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is then converted into illustrations, which the device displays on the user's device screen.

[0396] For example, if a user is having difficulty understanding a conversation while dining with a friend at a restaurant, the hearing aid will transcribe the conversation and display the analysis results along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," an illustration and key words related to the movie will be displayed on the screen.

[0397] Summarizing and illustrating online meetings

[0398] The system has the ability to collect audio from online meetings in real time, summarize it, and then illustrate and display it. Users participate in online meetings and connect their hearing aids to their PCs or smartphones to collect audio data from the meeting. The audio data is then converted into text and sent to a server. A generative AI model analyzes and summarizes the meeting content, and the summary is then converted into an illustration that is easier to understand visually. The results are then received by the terminal and displayed on the user's device.

[0399] For example, when a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, and the summary and illustrations are analyzed by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content and an illustration of the new product are displayed.

[0400] Incorporating an emotion engine

[0401] The system also incorporates an emotion engine that recognizes emotions from the user's facial expressions and voice, recognizing the user's emotional state in real time and adjusting other system functions accordingly.

[0402] For example, if a user expresses anxiety or irritation during a conversation, the emotion engine will recognize this and adjust the hearing aid settings appropriately. It will also change the illustrations and transcriptions provided based on the emotion to provide information that is best suited to the user's emotional state.

[0403] For example, if a user is feeling stressed during a business meeting, the emotion engine will detect that stress, summarize the conversation briefly, and display an illustration to help reduce stress. The tone and volume of the audio will also be adjusted, allowing the user to participate in the meeting in a more relaxed state.

[0404] The processing flow will be explained below.

[0405] Optimizing hearing aid settings

[0406] Processing Steps

[0407] Step 1:

[0408] The user puts on the hearing aid and launches the settings app on their smartphone or the hearing aid.

[0409] Step 2:

[0410] The user presses the "Optimize Settings" button in the app.

[0411] Step 3:

[0412] The device (hearing aid or smartphone) records the surrounding sound environment for a few seconds and collects auditory data such as volume and frequency spectrum.

[0413] Step 4:

[0414] The terminal transmits the collected auditory data to a server.

[0415] Step 5:

[0416] Based on the data received by the server, a generative AI model is run, which analyzes the auditory data and suggests optimal settings.

[0417] Step 6:

[0418] The server returns the configuration data including the analysis results to the terminal.

[0419] Step 7:

[0420] The device applies the received configuration data and automatically adjusts the audio settings of the hearing aid.

[0421] Step 8:

[0422] The user will see a "Setup complete" notification and be asked to review their listening progress and provide feedback.

[0423] Step 9:

[0424] If the user instructs fine adjustment as needed, the setting data is sent to the server again and adjustments are made using the same process.

[0425] Fall risk detection and prevention

[0426] Processing Steps

[0427] Step 1:

[0428] The user wears the hearing aid and goes about their daily activities.

[0429] Step 2:

[0430] The device (camera and sensors built into the hearing aid) collects the user's behavior data in real time.

[0431] Step 3:

[0432] The operating data collected by the device is temporarily stored and an initial analysis is performed using an anomaly detection algorithm.

[0433] Step 4:

[0434] The device sends the analysis results to the server and requests a detailed analysis.

[0435] Step 5:

[0436] Based on the data received by the server, a generative AI model is run to assess the risk of falling.

[0437] Step 6:

[0438] The server returns the results of the risk assessment to the terminal.

[0439] Step 7:

[0440] If the device is judged to be at high risk of falling, it will issue an alert to the user via voice or vibration to warn them.

[0441] Step 8:

[0442] If the user falls, the device will detect the fall and automatically send a text message or phone alert to pre-defined emergency contacts.

[0443] Transcription and illustration of conversation content

[0444] Processing Steps

[0445] Step 1:

[0446] The user puts on the hearing aid and begins a conversation.

[0447] Step 2:

[0448] The device (hearing aid) collects conversational audio in real time.

[0449] Step 3:

[0450] The device temporarily stores the voice data and performs initial voice recognition.

[0451] Step 4:

[0452] The terminal transmits the converted text data to the server.

[0453] Step 5:

[0454] Based on the text data received by the server, a generative AI model is run to analyze keywords and important points.

[0455] Step 6:

[0456] Based on the analysis results, the server converts the content into easy-to-understand illustrations.

[0457] Step 7:

[0458] The server sends the generated illustrations and transcription data back to the device.

[0459] Step 8:

[0460] The device (hearing aid or connected smartphone) displays the transcription results and illustrations on the user's screen.

[0461] Summarizing and illustrating online meetings

[0462] Processing Steps

[0463] Step 1:

[0464] A user joins an online meeting.

[0465] Step 2:

[0466] The user connects the conferencing app on their PC or smartphone to their hearing aid.

[0467] Step 3:

[0468] The device (hearing aid or PC / smartphone) collects the conference audio in real time.

[0469] Step 4:

[0470] The device transcribes the audio data and sends the generated text data to the server.

[0471] Step 5:

[0472] Based on the text data received by the server, a generative AI model is run to generate a summary of the meeting contents.

[0473] Step 6:

[0474] Based on the summary results, the server illustrates important points and keywords.

[0475] Step 7:

[0476] The server sends the summary and illustrations back to the device.

[0477] Step 8:

[0478] The device (hearing aid or connected smartphone) displays a summary and illustrations on the user's screen, allowing the user to efficiently understand the meeting content.

[0479] Incorporating an emotion engine

[0480] Processing Steps

[0481] Step 1:

[0482] The user wears the hearing aid and participates in daily activities, conversations, and online meetings.

[0483] Step 2:

[0484] The camera and microphone on the device (hearing aid or smartphone) collect the user's facial expressions and voice in real time.

[0485] Step 3:

[0486] The device uses the data collected to run an emotion engine and recognize the user's emotions.

[0487] Step 4:

[0488] The device transmits the recognized emotion data to the server.

[0489] Step 5:

[0490] The server runs a generative AI model based on the emotional data and analyzes the optimal hearing aid settings and information delivery methods.

[0491] Step 6:

[0492] The server returns the analysis results to the device.

[0493] Step 7:

[0494] The device adjusts the hearing aid settings and provides different illustrations and transcriptions depending on the user's emotional state.

[0495] Step 8:

[0496] Users can receive hearing aid settings and information that are optimized for their emotional state.

[0497] Example 2

[0498] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0499] While modern hearing aids offer a wide range of functions, they face challenges in adapting to individual user needs and circumstances. In particular, they lack the ability to automatically adjust settings in response to environmental changes, or to recognize the user's emotions and behavior in real time and respond appropriately. Furthermore, technologies for making online meetings and everyday conversations easier to understand are limited. This makes it difficult to provide effective support for the hearing impaired and elderly to live comfortably.

[0500] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting auditory data of the user, means for transmitting the collected auditory data to the server, means for analyzing the auditory data using an AI model generated by the server and proposing optimal hearing aid device settings, means for automatically adjusting hearing aid device settings based on the analysis results, means for collecting user movement data in real time, means for transmitting the collected movement data to the server, means for assessing a fall risk using an AI model generated by the server, means for alerting the user when a high risk of falling is detected, means for sending an alert to an emergency contact when a fall occurs, means for collecting conversation voice in real time, means for converting the collected voice data into text data, means for transmitting the text data to the server, and means for converting the collected voice data into text data using an AI model generated by the server. The system includes means for analyzing audio data from online meetings and extracting important points, means for converting content including important points into easy-to-understand images, means for displaying the converted images and transcription data to the user, means for collecting the user's emotional state, means for transmitting the collected emotional data to a server, means for analyzing the emotional data using an AI model generated by the server and adjusting other system functions, means for providing information according to the emotions, means for collecting audio data from online meetings in real time, means for converting the collected audio data into text data, means for transmitting the text data to a server, means for analyzing the text data using an AI model generated by the server and summarizing it, means for converting the summarized content into easy-to-understand images, and means for displaying the converted images and transcription data to the user. This enables automatic adaptation to the environment and situation, advance detection and prevention of fall risks, support for understanding conversations and online meeting content, and appropriate responses based on the user's emotions.

[0501] "Hearing data" refers to data that includes various parameters related to hearing, such as volume and frequency spectrum.

[0502] A "hearing assistive device" is a device worn on the ear that amplifies or converts audio signals to assist the user's hearing.

[0503] "Motion Data" is data that includes a user's physical movements and location information and is collected by sensors.

[0504] The "fall risk" is an evaluation value of the possibility that the user will fall, and is calculated based on the motion data.

[0505] "Audio Data" refers to audio signals collected by a microphone and converted into digital form.

[0506] "Text data" is data that has been converted from voice data into characters or words and is expressed as a sentence.

[0507] "Important points" are information or keywords that are particularly important to the user and are extracted from the text data.

[0508] An "image" is a figure or picture that is generated to make information easier to understand visually.

[0509] "Emotion data" is data that indicates the emotional state of the user that can be inferred from facial expressions, tone of voice, and the like.

[0510] An "online conference" is a conference held between remote locations in real time via the Internet.

[0511] A "summary" is a concise summary of the essence or important points extracted from text or audio data.

[0512] The present invention is a hearing aid system that utilizes generative AI models and incorporates user emotion recognition. This system aims to solve the challenges faced by modern hearing aids. Specific implementations of each function are described in detail below.

[0513] Hearing data collection and setting optimization

[0514] When a user puts on their hearing aid and launches the settings app, the device records the surrounding sound environment for a few seconds, collecting hearing data such as volume and frequency spectrum. This data is sent wirelessly (via Wi-Fi or mobile data) to a server. The server analyzes the data using a generative AI model and recommends optimal hearing aid settings. The analysis results are sent back to the device, which then automatically adjusts the hearing aid settings.

[0515] Example: If a user points out that they have difficulty hearing voices in a movie theater, the device analyzes the sound environment in the theater, and the AI ​​model suggests optimal volume and filter settings. The device then applies these suggestions to the hearing aid, and the user can check the sound quality.

[0516] Fall risk detection and prevention

[0517] The user's movement data is collected in real time by cameras and sensors built into the hearing aid device. This data is sent to a server, where a generative AI model analyzes the movement data and assesses the risk of falling. If a high risk of falling is detected, the server sends an alert to the device, which then issues an audio or vibration alert to the user. If a fall actually occurs, the hearing aid device automatically sends an alert to emergency contacts.

[0518] Example: If a user loses balance while descending stairs, the device will immediately sound an alarm to alert the user, and if the user still falls, an alert will be automatically sent to emergency contacts.

[0519] Transcription and illustration of conversation content

[0520] When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio in real time and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is then converted into an illustration, which the device displays on the user's device screen.

[0521] Example: If a user is having difficulty understanding the conversation while dining with friends at a restaurant, the device will transcribe the conversation and display the analysis results along with illustrations on the smartphone.

[0522] Example prompt: "I saw a movie yesterday," followed by "Show me an illustration and key words related to the movie."

[0523] Summarizing and illustrating online meetings

[0524] Users participate in online meetings and connect their hearing aids to their PCs or smartphones to collect audio data from the meeting. This data is converted into text and sent to a server. A generative AI model analyzes the meeting content and converts the summary into an illustration. The result is sent to the user's device and displayed.

[0525] Example: When a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, and the summary and illustrations are analyzed by the server and displayed on the smartphone.

[0526] Example prompt: "Show me a summary and related illustration of your new product launch plan."

[0527] Built-in emotion recognition engine

[0528] The system incorporates an emotion engine that recognizes the user's emotional state in real time from facial expressions and vocal tone. This emotion engine adjusts the system settings appropriately based on the user's emotions. The emotion engine provides information and adjusts settings appropriate to the user's emotions based on other system functions.

[0529] Example: If a user is feeling stressed during a business meeting, the emotion engine will recognize the stress, summarize the conversation, and display an illustration to help the user reduce stress. It will also adjust the tone and volume of the audio.

[0530] Example prompt: "The user is feeling stressed. Please provide information and images that will encourage relaxation."

[0531] The system of the present invention has the above-mentioned functions and is intended to optimize the user's auditory experience in various situations while improving safety and information provision.

[0532] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0533] Optimizing hearing aid settings

[0534] Step 1:

[0535] The user puts on the hearing aid and launches the settings app on their smartphone.

[0536] Input: The user puts on the hearing aid and touches it to launch the app.

[0537] Output: The app's home screen is displayed.

[0538] Specific behavior: The user selects and taps the Settings app icon on the smartphone's home screen.

[0539] Step 2:

[0540] When the user presses the "Optimize Settings" button in the app, the device begins recording the surrounding sound environment.

[0541] Input: User taps the "Optimize Settings" button.

[0542] Output: Sound environment data (volume, frequency spectrum) is collected.

[0543] What it does: The device's microphone is activated and collects the sound environment for a few seconds.

[0544] Step 3:

[0545] The device transmits the collected sound environment data to the server.

[0546] Input: Collected sound environment data.

[0547] Output: Sound environment data sent to the server.

[0548] What happens: Your device uploads data using Wi-Fi or mobile data.

[0549] Step 4:

[0550] The server analyzes the data using a generative AI model and suggests optimal settings for the hearing aid.

[0551] Input: Sound environment data sent to the server.

[0552] Output: Optimal setting parameters (volume settings and filter settings).

[0553] What it does: The server runs a Python script, analyzes data with an AI model, and calculates configuration parameters.

[0554] Step 5:

[0555] The server sends the analysis results back to the device, which then automatically adjusts the hearing aid settings.

[0556] Input: The optimal configuration parameters sent by the server.

[0557] Output: The applied hearing aid settings.

[0558] Specific operation: The device applies the received configuration parameters to the hearing aid system.

[0559] Fall risk detection and prevention

[0560] Step 1:

[0561] Cameras and sensors built into the device collect user behavior data in real time.

[0562] Input: User movement data from cameras and sensors.

[0563] Output: Collected behavioral data.

[0564] How it works: The hearing aid's accelerometer and gyroscope sensors continuously monitor and record the user's movements.

[0565] Step 2:

[0566] This motion data is sent to the server.

[0567] Input: Collected behavioral data.

[0568] Output: The operation data sent to the server.

[0569] What happens: Your device compresses the data and uploads it to the server over a secure connection.

[0570] Step 3:

[0571] The server uses the generated AI model to analyze movement data and assess the risk of falling.

[0572] Input: The operational data sent to the server.

[0573] Output: Fall risk assessment score.

[0574] Specific behavior: The server identifies patterns and anomalous behavior and uses a scoring algorithm to assess risk.

[0575] Step 4:

[0576] If a high risk of falling is detected, the server sends an alert to the device, which then warns the user with sound or vibration.

[0577] Input: Fall risk assessment score.

[0578] Output: Warning alert (audio and vibration) by the device.

[0579] What happens: The device plays a specific audio file and vibrates.

[0580] Step 5:

[0581] If a fall occurs, the hearing aid device will send an alert to emergency contacts.

[0582] Input: Actual fall data.

[0583] Output: Alert sent to emergency contacts.

[0584] What happens: Your device will send an SMS or automated call to your emergency contacts.

[0585] Transcription and illustration of conversation content

[0586] Step 1:

[0587] When a user starts a conversation, the terminal collects the conversation voice in real time.

[0588] Input: Audio data from a conversation.

[0589] Output: Collected audio data.

[0590] How it works: The device's microphone records the conversation and processes the data in real time.

[0591] Step 2:

[0592] The device transcribes the collected voice data and converts it into text data.

[0593] Input: Collected audio data.

[0594] Output: Transcribed text data.

[0595] Specific operation: The device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the voice data into text.

[0596] Step 3:

[0597] The converted text data is sent to the server.

[0598] Input: Transcribed text data.

[0599] Output: The text data sent to the server.

[0600] What happens: Your device uploads data to the server using Wi-Fi or mobile data.

[0601] Step 4:

[0602] The server's generative AI model analyzes the text data and extracts key points.

[0603] Input: Text data sent to the server.

[0604] Output: Extracted key points.

[0605] What it does: The server uses NLP (Natural Language Processing) tools to identify and summarize the main points of the text.

[0606] Step 5:

[0607] The extracted content is converted into an illustration, which the terminal displays on the user's device screen.

[0608] Input: Extracted key points.

[0609] Output: The generated illustration and the display data that contains it.

[0610] Specific operation: The server sends the generated illustration to the device, which then displays the illustration on the user's smartphone or tablet.

[0611] Summarizing and illustrating online meetings

[0612] Step 1:

[0613] The user joins an online meeting and connects their hearing aid to their PC or smartphone.

[0614] Input: Audio data from the conference.

[0615] Output: Collected audio data.

[0616] How it works: The microphone on your PC or smartphone picks up the conference audio and transmits the audio data to your hearing aids in real time.

[0617] Step 2:

[0618] The hearing aid converts the audio data of the meeting into text and sends it to the server.

[0619] Input: Collected audio data.

[0620] Output: Transcribed text data.

[0621] What it does: The hearing aid uses a speech recognition API to convert speech into text and sends that data to the server.

[0622] Step 3:

[0623] The server uses a generative AI model to generate the summary.

[0624] Input: The text data sent.

[0625] Output: Summarized text data.

[0626] What it does: The server uses NLP tools to analyze the meeting content and summarize the key points.

[0627] Step 4:

[0628] The summary results are converted into illustrations, which the terminal displays on the user's device screen.

[0629] Input: Summarized text data.

[0630] Output: The generated illustration and the display data that contains it.

[0631] Specific operation: The server sends the generated illustration to the device, which then displays the illustration on the user's smartphone or tablet.

[0632] Built-in emotion recognition engine

[0633] Step 1:

[0634] The device collects the user's facial expressions and voice tone and analyzes the emotional data.

[0635] Input: User's facial expression data, voice tone data.

[0636] Output: Collected emotion data.

[0637] How it works: The device's camera and microphone capture the user's facial expressions and tone of voice, which are then analyzed by emotion recognition software.

[0638] Step 2:

[0639] The collected emotion data is sent to a server.

[0640] Input: Data from emotion recognition.

[0641] Output: Emotion data sent to the server.

[0642] Specific operation: The device uploads emotion data to the server using Wi-Fi or mobile data.

[0643] Step 3:

[0644] The server uses a generative AI model to analyze the emotional data and adjust the system settings.

[0645] Input: The submitted emotion data.

[0646] Output: Adjusted system configuration parameters.

[0647] How it works: The server analyzes emotional data using an AI model and optimally sets the tone of voice and the way information is presented.

[0648] Step 4:

[0649] Information is provided according to emotions, and the device adjusts the screen display and audio appropriately.

[0650] Input: Adjusted system configuration parameters.

[0651] Output: Adjusted informational and audio settings.

[0652] Specific operation: The device provides information according to the user's emotions, displays information and illustrations to encourage relaxation, and adjusts the tone of the voice.

[0653] (Application example 2)

[0654] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0655] Conventional hearing aid systems often only provide simple hearing aid functions related to the user's hearing, making them inappropriate for specific environments such as factories. They also lack features that reduce the risk of falls in work environments and make it easier to understand the work being done. The present invention aims to solve these problems and provide an advanced hearing aid system that can be used in environments such as factories.

[0656] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the user's hearing data, means for transmitting the collected hearing data to the server, means for analyzing the hearing data using an AI model generated by the server and proposing optimal hearing aid settings, means for automatically adjusting the hearing aid settings based on the analysis results, and means for recording and analyzing the sound environment in a factory environment and proposing sound setting adjustments. This makes it possible to provide an optimal hearing environment even in a factory and improve work efficiency.

[0657] "Auditory data" refers to audio information related to the user's hearing, and specifically refers to collected information on sound waves such as surrounding environmental sounds and human voices.

[0658] "Server" refers to the central control unit for analyzing collected data and running generative AI models.

[0659] A "generative AI model" is a data analysis model that uses artificial intelligence algorithms executed on a server to evaluate and analyze the user's auditory and behavioral data.

[0660] "Analysis results" refers to the results of data analyzed by the generative AI model, including suggestions for hearing aid settings and fall risk assessment results.

[0661] "Hearing aid settings" refers to the operating parameters of a hearing aid to assist the user's hearing, such as adjusting the volume or frequency spectrum.

[0662] "Motion data" refers to information about a user's physical movements and posture, and is collected using sensors and cameras.

[0663] The "fall risk" refers to the evaluation result of the possibility that the user will fall, and is calculated based on the motion data.

[0664] "Emergency contacts" refers to contacts to be notified in the event of an emergency such as a fall, such as the phone numbers of medical personnel or family members.

[0665] "Conversational voice" refers to voice information in a conversation between a user and another person, and is collected using a microphone or the like.

[0666] "Text data" refers to information obtained by converting voice data into text, such as transcribed conversation content.

[0667] "Illustration" refers to a picture or diagram drawn to visually show content, including text data.

[0668] The present invention is an innovative factory work environment optimization system that utilizes a generative AI model and incorporates a user emotion recognition function. Specific embodiments for carrying out the present invention will be described in detail below.

[0669] 1. System Configuration

[0670] The system mainly consists of the following components:

[0671] Audio collection device (microphone, etc.)

[0672] Motion collection devices (cameras, sensors, etc.)

[0673] server

[0674] User device (smartphone, tablet, etc.)

[0675] 2. Program processing content

[0676] The system performs processing using the following hardware and software.

[0677] Recording and analyzing sound environments

[0678] Hardware used: Microphone

[0679] Software used: SoundAnalysis

[0680] The user device collects the sound environment in the factory using a microphone and transmits the data in real time to a server, where a generative AI model analyzes the audio data and suggests optimal volume and frequency spectrum settings.

[0681] Fall risk detection

[0682] Hardware used: Camera, sensor

[0683] Software used: Motion analysis algorithm

[0684] Cameras and sensors monitor the user's movements in real time and send the collected data to a server. The generative AI model analyzes this movement data and assesses the risk of falling. If a high risk is detected, the device will issue an alert to the user.

[0685] Summary and illustration of work

[0686] Hardware used: Microphone

[0687] Software used: SpeechToText, TextToIllustration

[0688] A microphone collects the user's conversations while they are working and sends them to the server. The generative AI model converts the voice data into text and extracts key points. The results are converted into illustrations and displayed on the user's device.

[0689] Improving work efficiency through emotion recognition

[0690] Hardware used: Camera, microphone

[0691] Software used: EmotionRecognition

[0692] The system uses cameras and microphones to collect the user's facial expressions and voice, which are then sent to a server. The generative AI model analyzes this data to recognize the user's emotional state. Based on the recognized emotions, the system automatically adjusts its work settings to reduce stress and fatigue.

[0693] 3. Specific Examples

[0694] Optimizing the sound environment

[0695] For example, if a machine in a factory makes an abnormal sound, a microphone picks up the sound and sends it to a server. The generative AI model analyzes the audio data, determines that the sound is a motor malfunction, and suggests optimal volume and filter settings. The user's device then applies the settings and checks the quality of the sound.

[0696] Fall risk detection

[0697] If the user is about to lose balance while working, the sensor detects abnormal movements and immediately issues an alert to the user, and if the fall still occurs, an alert is automatically sent to emergency contacts.

[0698] Summary of work

[0699] When a user starts talking, a microphone collects the audio and sends it to a server. The generative AI model converts the conversation into text, extracts key points, and converts them into illustrations. The illustrations and text are then displayed on the user's device, making it easier for the user to understand the task at hand.

[0700] 4. Example prompt (text format)

[0701] "Analyze the sound data below and suggest optimal earplug settings. If there is additional anxiety or irritation, generate and display appropriate illustrations."

[0702] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0703] Step 1:

[0704] The user terminal collects the sound environment in the factory using a microphone.

[0705] Input: Environmental sounds inside the factory

[0706] Output: Collected audio data

[0707] How it works: Microphones record the sounds of machinery and alarms generated within the factory in real time and store the data in digital format.

[0708] Step 2:

[0709] The collected voice data is sent to a server.

[0710] Input: Collected audio data

[0711] Output: Audio data sent to the server

[0712] Specific operation: The user device uploads the saved voice data to the server using Wi-Fi or wired communication.

[0713] Step 3:

[0714] The server uses a generative AI model to analyze the audio data and suggest optimal volume and frequency spectrum settings.

[0715] Input: Audio data sent to the server

[0716] Output: Optimal volume and frequency spectrum settings

[0717] How it works: A generative AI model on a server analyzes audio data and uses detectors for abnormal sounds and specific frequencies to calculate optimal acoustic settings.

[0718] Step 4:

[0719] The server automatically adjusts the hearing aid settings based on the analysis results.

[0720] Input: Optimal volume and frequency spectrum settings

[0721] Output: Adjusted hearing aid settings

[0722] Specific operation: The server sends adjusted setting information to the user device, and the user device updates the hearing aid settings in real time.

[0723] Step 5:

[0724] Cameras and sensors collect user behavior data in real time.

[0725] Input: User action

[0726] Output: Collected behavioral data

[0727] Specific movements: Cameras and sensors capture the user's movements, such as gait and posture, and store the data in digital form.

[0728] Step 6:

[0729] The collected operation data is sent to a server.

[0730] Input: Collected behavioral data

[0731] Output: Operational data sent to the server

[0732] Specific operation: The user terminal uploads operation data to the server using the communication infrastructure.

[0733] Step 7:

[0734] The server uses a generative AI model to analyze movement data and assess fall risk.

[0735] Input: Operational data sent to the server

[0736] Output: Fall risk assessment results

[0737] Specific movements: A generative AI model analyzes movement data and applies algorithms to detect abnormal movements and assess fall risk.

[0738] Step 8:

[0739] If a high risk of falling is detected, an alert will be issued to the user.

[0740] Input: Fall risk assessment results

[0741] Output: A warning alert to the user

[0742] Specific operation: The device will issue a warning sound or vibration to the user in real time to alert them.

[0743] Step 9:

[0744] A microphone collects the conversational audio and transmits it to a server.

[0745] Input: Conversational audio

[0746] Output: Conversational audio data sent to the server

[0747] What it does: The microphone records the audio of the conversation, stores it digitally, and sends it to a server.

[0748] Step 10:

[0749] The server converts the voice data into text data.

[0750] Input: Conversational audio data sent to the server

[0751] Output: Text data

[0752] What it does: It uses a generative AI model to transcribe audio data and convert it into text format.

[0753] Step 11:

[0754] The server analyzes the text data and extracts important points.

[0755] Input: Text data

[0756] Output: Key points excerpt

[0757] How it works: The generative AI model analyzes the conversation, identifying and extracting keywords and key phrases.

[0758] Step 12:

[0759] The extracted important points are converted into illustrations and displayed on the user's device.

[0760] Input: Key points excerpt

[0761] Output: Visually easy-to-understand illustrations

[0762] Specific operation: The generative AI model creates an illustration based on key points and displays it on the user's device.

[0763] Step 13:

[0764] The camera and microphone collect the user's facial expressions and voice and transmit them to the server.

[0765] Input: User's facial expression and voice data

[0766] Output: Facial expression and voice data sent to the server

[0767] How it works: The camera and microphone record the user's facial expressions and voice in real time and send them digitally to a server.

[0768] Step 14:

[0769] The server uses an emotion recognition algorithm to recognize the user's emotional state.

[0770] Input: Facial expression and voice data sent to the server

[0771] Output: User's emotional state

[0772] How it works: The generative AI model analyzes the user's facial and voice data to identify emotions.

[0773] Step 15:

[0774] The system adjusts its settings based on the user's emotional state and provides the user with information accordingly.

[0775] Input: User's emotional state

[0776] Output: Adjusted system settings and information provided

[0777] How it works: The server calculates the optimal settings based on the user's emotional state and sends that information to the user's device. The device then adjusts the system settings in real time and provides information based on the emotion.

[0778] The above are the specific processing steps.

[0779] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0780] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0781] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0782] [Second embodiment]

[0783] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0784] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0785] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0786] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0787] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0788] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0789] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0790] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0791] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0792] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0793] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0794] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0795] The present invention is an innovative hearing aid system that utilizes generative AI models and is equipped with multiple functions. Specific embodiments are described below.

[0796] Optimizing hearing aid settings

[0797] This hearing aid system has the ability to automatically optimize hearing aid settings based on the user's hearing environment by simply putting on the hearing aid and launching the settings app. When the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal hearing aid settings. The analysis results are sent back to the device, which automatically adjusts the hearing aid settings. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[0798] For example, if a user finds it difficult to hear people during a meeting, they can optimize their hearing aid settings. The hearing aid device then analyzes the sound environment of the meeting, and the AI ​​model proposes optimal volume and filter settings. The hearing aid device then applies the proposed settings, and the user can confirm the actual effect.

[0799] Fall risk detection and prevention

[0800] The system detects and prevents the risk of falls by collecting user movement data in real time and sending it to a server. Cameras and sensors built into the hearing aid device monitor the user's movements and collect movement data. This data is temporarily stored in the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends an alert to the device, which then alerts the user via audio and vibration. Additionally, if the user actually falls, the hearing aid device immediately sends an alert to emergency contacts.

[0801] For example, if a user is about to lose their balance while descending stairs, the hearing aid device will immediately sound an alarm to alert the user, and if they still fall, an alert will be sent to their family or caregiver.

[0802] Transcription and illustration of conversation content

[0803] The system collects conversational audio in real time, transcribes it, and converts it into illustrations to help users understand the content of the conversation. When a user puts on their hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[0804] For example, if a user is having difficulty understanding the conversation while dining with a friend at a restaurant, the hearing aid will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will appear on the screen.

[0805] Summarizing and illustrating online meetings

[0806] The system collects audio from online meetings in real time, converts the content into text, summarizes it, and displays it as illustrations. Users participate in online meetings, and their hearing aids connect to their PCs or smartphones to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen by the hearing aid terminal or a connected device.

[0807] For example, when a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed, along with an illustration of the new product.

[0808] The above is a detailed description of the mode for carrying out the invention.

[0809] The processing flow will be explained below.

[0810] Optimizing hearing aid settings

[0811] Processing Steps

[0812] Step 1:

[0813] The user puts on the hearing aid and launches the settings app on their smartphone or the hearing aid.

[0814] Step 2:

[0815] The user presses the "Optimize Settings" button in the app.

[0816] Step 3:

[0817] The device (hearing aid or smartphone) records the surrounding sound environment for a few seconds and collects auditory data such as volume and frequency spectrum.

[0818] Step 4:

[0819] The terminal transmits the collected auditory data to a server.

[0820] Step 5:

[0821] Based on the data received by the server, a generative AI model is run, which analyzes the auditory data and suggests optimal settings.

[0822] Step 6:

[0823] The server returns the configuration data including the analysis results to the terminal.

[0824] Step 7:

[0825] The device applies the received configuration data and automatically adjusts the audio settings of the hearing aid.

[0826] Step 8:

[0827] The user will see a "Setup complete" notification and be asked to review their listening progress and provide feedback.

[0828] Step 9:

[0829] If the user instructs fine adjustment as needed, the setting data is sent to the server again and adjustments are made using the same process.

[0830] Fall risk detection and prevention

[0831] Processing Steps

[0832] Step 1:

[0833] The user wears the hearing aid and goes about their daily activities.

[0834] Step 2:

[0835] The device (camera and sensors built into the hearing aid) collects the user's behavior data in real time.

[0836] Step 3:

[0837] The operating data collected by the device is temporarily stored and an initial analysis is performed using an anomaly detection algorithm.

[0838] Step 4:

[0839] The device sends the analysis results to the server and requests a detailed analysis.

[0840] Step 5:

[0841] Based on the data received by the server, a generative AI model is run to assess the risk of falling.

[0842] Step 6:

[0843] The server returns the results of the risk assessment to the terminal.

[0844] Step 7:

[0845] If the device is judged to be at high risk of falling, it will issue an alert to the user via voice or vibration to warn them.

[0846] Step 8:

[0847] If the user falls, the device will detect the fall and automatically send a text message or phone alert to pre-defined emergency contacts.

[0848] Transcription and illustration of conversation content

[0849] Processing Steps

[0850] Step 1:

[0851] The user puts on the hearing aid and begins a conversation.

[0852] Step 2:

[0853] The device (hearing aid) collects conversational audio in real time.

[0854] Step 3:

[0855] The device temporarily stores the voice data and performs initial voice recognition.

[0856] Step 4:

[0857] The terminal transmits the converted text data to the server.

[0858] Step 5:

[0859] Based on the text data received by the server, a generative AI model is run to analyze keywords and important points.

[0860] Step 6:

[0861] Based on the analysis results, the server converts the content into easy-to-understand illustrations.

[0862] Step 7:

[0863] The server sends the generated illustrations and transcription data back to the device.

[0864] Step 8:

[0865] The device (hearing aid or connected smartphone) displays the transcription results and illustrations on the user's screen.

[0866] Summarizing and illustrating online meetings

[0867] Processing Steps

[0868] Step 1:

[0869] A user joins an online meeting.

[0870] Step 2:

[0871] The user connects the conferencing app on their PC or smartphone to their hearing aid.

[0872] Step 3:

[0873] The device (hearing aid or PC / smartphone) collects the conference audio in real time.

[0874] Step 4:

[0875] The device transcribes the audio data and sends the generated text data to the server.

[0876] Step 5:

[0877] Based on the text data received by the server, a generative AI model is run to generate a summary of the meeting contents.

[0878] Step 6:

[0879] Based on the summary results, the server illustrates important points and keywords.

[0880] Step 7:

[0881] The server sends the summary and illustrations back to the device.

[0882] Step 8:

[0883] The device (hearing aid or connected smartphone) displays a summary and illustrations on the user's screen, allowing the user to efficiently understand the meeting content.

[0884] Example 1

[0885] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0886] Conventional hearing aid systems have difficulty in properly adjusting to the user's individual hearing environment and operating conditions. Furthermore, proper settings according to the environment are often performed manually, placing a burden on the user. Furthermore, functions for detecting the risk of falls and assisting in understanding conversation content are lacking, making them insufficient for improving the user's safety and quality of daily life. A comprehensive system that solves these problems is needed.

[0887] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0888] In this invention, the server includes means for analyzing the user's hearing data and proposing optimal hearing aid settings, means for analyzing the user's motion data and assessing the risk of falling, means for analyzing conversational audio, extracting key points and illustrating them, and means for analyzing conference audio, creating summaries and illustrating them, thereby improving the user's hearing environment and safety, and enabling them to understand the content of conversations and conferences.

[0889] "Hearing data" refers to data related to the user's hearing, such as volume and frequency spectrum obtained through a hearing aid.

[0890] "Motion data" is data that includes information about the user's body movements and posture.

[0891] A "server" is a computer system that works in conjunction with hearing aid terminals via a network to analyze and process data.

[0892] A "generative AI model" is a machine learning algorithm that analyzes collected data and generates optimal suggestions and results.

[0893] "Analysis" is the process of investigating and examining collected data in detail using specialized methods to derive useful information and conclusions.

[0894] "Hearing aid settings" are settings that adjust various parameters of the hearing aid (e.g., volume, frequency filters, etc.).

[0895] "Fall risk" is an assessment that indicates the degree of possibility that the user will fall.

[0896] An "alert" is a notification that warns or alerts the user.

[0897] "Conversational voice" refers to the voice that a user utters while communicating with others.

[0898] "Text data" refers to data obtained by converting voice data into text format.

[0899] "Important points" are parts of a conversation or meeting that are judged to be particularly important.

[0900] An "illustration" is an image that visually represents an important point or summary.

[0901] A "summary" is a concise summary of long information such as that from a meeting.

[0902] The present invention is an innovative hearing assistance system that utilizes a generative AI model and is equipped with multiple functions. Specific embodiments are described below.

[0903] Optimizing hearing aid settings

[0904] This hearing assistance system has the function of automatically optimizing settings based on the hearing environment when the user wears the hearing assistance device and launches the settings app on their smartphone. Specifically, when the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal settings. The analysis results are sent back to the device, which automatically adjusts the settings of the hearing assistance device. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[0905] For example, if a user finds it difficult to hear people during a meeting, they can optimize their settings. The device will then analyze the sound environment of the meeting, and the AI ​​model will suggest optimal volume and filter settings. The device will then apply the suggested settings, and the user can check the actual effect.

[0906] Fall risk detection and prevention

[0907] The system has the ability to detect and prevent the risk of falls by collecting user movement data in real time and sending it to a server. The device's built-in cameras and sensors monitor the user's movements and collect movement data. This data is temporarily stored on the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends a warning alert to the device, and the device alerts the user via audio and vibration. Additionally, if the user actually falls, the device immediately sends an alert to emergency contacts.

[0908] For example, if a user is about to lose balance while descending stairs, the device will immediately sound an alarm to warn the user, and if the user still falls, an alert will be sent to a family member or caregiver.

[0909] Transcription and illustration of conversation content

[0910] The system collects conversational audio in real time, transcribes it, and converts it into easy-to-understand illustrations. When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[0911] For example, if a user is having difficulty understanding the conversation while dining with friends at a restaurant, the hearing aid device will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will be displayed on the screen.

[0912] Summarizing and illustrating online meetings

[0913] The system has the ability to collect audio from online meetings in real time, convert the content into text, summarize it, and display it as illustrations. Users participate in online meetings, and the hearing aid connects with a PC or smartphone to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen via their terminal or a connected device.

[0914] For example, when a user participates in a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed on the screen, along with an illustration of the new product.

[0915] Prompt Sentence Examples

[0916] "Please suggest the best assistive hearing device settings for me during the meeting."

[0917] "Send an alert when the user loses balance while descending stairs."

[0918] "Please transcribe the conversation in the restaurant and show it in illustrations."

[0919] "Please summarize and illustrate the contents of the Zoom meeting."

[0920] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0921] Optimizing hearing aid settings

[0922] Step 1:

[0923] The user launches the Settings app on their smartphone and presses the "Optimize Settings" button.

[0924] Input: User actions

[0925] Output: Triggers configuration optimization

[0926] At this point, the setting optimization process is initiated by the user.

[0927] Step 2:

[0928] The device will use the built-in microphone to record the surrounding sound environment for a few seconds.

[0929] Input: Trigger settings optimization

[0930] Output: Sound environment data

[0931] Specifically, the device captures ambient sounds during the meeting.

[0932] Step 3:

[0933] The device extracts auditory data such as volume and frequency spectrum.

[0934] Input: Sound environment data

[0935] Output: Auditory data

[0936] The collected sound environment data is analyzed and the necessary auditory data is extracted.

[0937] Step 4:

[0938] The terminal compresses the auditory data and prepares it for transmission to the server.

[0939] Input: Auditory data

[0940] Output: Compressed data

[0941] Data such as volume and frequency spectrum is compressed and converted into a format that is sent to the server.

[0942] Step 5:

[0943] The terminal transmits the compressed data to the server.

[0944] Input: Compressed data

[0945] Output: Data on the server

[0946] Data is sent from the terminal to the server.

[0947] Step 6:

[0948] The server unpacks the received data and begins analyzing it.

[0949] Input: Compressed data

[0950] Output: Expanded data

[0951] Extract the data and prepare it for analysis.

[0952] Step 7:

[0953] An AI model generated on the server analyzes the deployment data and calculates the optimal settings.

[0954] Input: Expanded data

[0955] Output: Optimal Settings

[0956] An AI model examines the data and determines the optimal hearing aid settings for the user.

[0957] Step 8:

[0958] The server returns the analysis results to the device.

[0959] Input: Optimal Settings

[0960] Output: Configuration data on the terminal

[0961] The server communicates the optimal settings to the device.

[0962] Step 9:

[0963] The settings of the hearing aid device are automatically adjusted based on the analysis results received by the device.

[0964] Input: Setting data

[0965] Output: Calibrated hearing aids

[0966] Your device automatically changes the settings of your hearing aid.

[0967] Step 10:

[0968] The user will be notified that the setup is complete.

[0969] Input: Adjusted hearing aid

[0970] Output: Notification

[0971] The user is notified when the settings are complete, and can make fine adjustments as needed.

[0972] Fall risk detection and prevention

[0973] Step 1:

[0974] Cameras and sensors built into the device monitor and collect information about the user's behavior.

[0975] Input: User action

[0976] Output: Operation data

[0977] Obtain user behavior data in real time.

[0978] Step 2:

[0979] The operating data collected by the device is temporarily saved and an initial analysis is performed.

[0980] Input: Operation data

[0981] Output: Initial analysis results

[0982] An initial analysis is performed using an anomaly detection algorithm on the device.

[0983] Step 3:

[0984] The device sends the initial analysis results to the server.

[0985] Input: Initial analysis results

[0986] Output: Data on the server

[0987] The initial analysis results are sent to the server to prepare for detailed analysis.

[0988] Step 4:

[0989] The server expands the received data and performs a detailed fall risk assessment using a generative AI model.

[0990] Input: Initial analysis results

[0991] Output: Detailed risk assessment

[0992] The server performs detailed analysis and assesses the risk of falling.

[0993] Step 5:

[0994] If a high risk of falling is detected, the server generates an alert to warn the user.

[0995] Input: Detailed risk assessment

[0996] Output: Alert

[0997] The server generates an alert and sends it to the device.

[0998] Step 6:

[0999] The device receives an alert and sends it to the user via sound or vibration.

[1000] Input: Alert

[1001] Output: Caution

[1002] The device will notify the user of the alert by sound or vibration.

[1003] Step 7:

[1004] If the user actually falls, the device will automatically make an emergency call.

[1005] Input: Fall detection

[1006] Output: Emergency alert

[1007] It detects if the user falls and sends an alert to emergency contacts.

[1008] Transcription and illustration of conversation content

[1009] Step 1:

[1010] A user wears a hearing aid and the terminal collects conversational audio.

[1011] Input: Conversational audio

[1012] Output: Audio data

[1013] To acquire a user's conversation voice in real time.

[1014] Step 2:

[1015] The device converts the collected voice data into text.

[1016] Input: Audio data

[1017] Output: Text data

[1018] Voice recognition software is used to convert voice data into text data.

[1019] Step 3:

[1020] Sends text data to the server.

[1021] Input: Text data

[1022] Output: Data on the server

[1023] The text data is sent to the server.

[1024] Step 4:

[1025] An AI model generated on the server analyzes the text data and extracts key points.

[1026] Input: Text data

[1027] Output: Key Points

[1028] An AI model is used to analyze text data and extract key points from the conversation.

[1029] Step 5:

[1030] The server converts the extracted key points into illustrations.

[1031] Input: Key Points

[1032] Output: illustration data

[1033] Convert key points into illustrations to make them easier to understand visually.

[1034] Step 6:

[1035] The device displays the converted illustration and transcription data on the user's screen.

[1036] Input: illustration data, transcription data

[1037] Output: What the user sees on their screen

[1038] The device displays the illustration and transcription results to provide the user.

[1039] Summarizing and illustrating online meetings

[1040] Step 1:

[1041] A user participates in an online conference, and a terminal collects the conference audio.

[1042] Input: Conference audio

[1043] Output: Audio data

[1044] Collect the user's online conference audio.

[1045] Step 2:

[1046] The terminal converts the collected voice data into text data.

[1047] Input: Audio data

[1048] Output: Text data

[1049] Use speech recognition software to convert meeting audio into text data.

[1050] Step 3:

[1051] Sends text data to the server.

[1052] Input: Text data

[1053] Output: Data on the server

[1054] The text data is sent to the server.

[1055] Step 4:

[1056] The AI ​​model generated on the server analyzes the text data and creates a summary.

[1057] Input: Text data

[1058] Output: Summary data

[1059] The AI ​​model analyzes the meeting content and generates a summary.

[1060] Step 5:

[1061] The server converts the summarized content into illustrations.

[1062] Input: Summary data

[1063] Output: illustration data

[1064] Convert summarized meeting content into visually easy-to-understand illustrations.

[1065] Step 6:

[1066] The terminal displays the converted illustration and summary data on the user's screen.

[1067] Input: illustration data, summary data

[1068] Output: What the user sees on their screen

[1069] The terminal displays the illustration and summary content for the user.

[1070] (Application example 1)

[1071] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1072] Conventional hearing aid systems are limited to analyzing voice and motion data and lack optimal functionality for real-world work environments and safety. Noisy factory environments, in particular, can make it difficult to hear voice instructions and increase the risk of workers falling. A method is needed to resolve these issues and improve work efficiency and safety.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1074] In this invention, the server

[1075] A means to analyze environmental sounds in the factory in real time and provide voice instructions to workers with optimal volume and sound quality,

[1076] A method for collecting worker movement data in real time using sensors and assessing the risk of falls.

[1077] A means to transcribe voice instructions and meeting content in real time and convert them into easy-to-understand illustrations for display,

[1078] This enables optimization of voice instructions according to the work environment, improvement of worker safety, and visualization of information.

[1079] "Auditory data" is acoustic information obtained through the user's ears, and is data that includes characteristics such as volume and frequency spectrum.

[1080] A "server" is a computer system on a network that collects, analyzes, and stores data.

[1081] An "AI model" is a model that uses machine learning algorithms to analyze data and derive optimal solutions to specific problems.

[1082] "Hearing aid settings" are various adjustment items for assisting the user's hearing, such as the volume and frequency filter settings of the hearing aid.

[1083] "Motion data" is data that includes information about the user's body movements and posture.

[1084] The "fall risk" is an evaluation value of the possibility that the user will fall, and is calculated based on the motion data.

[1085] A "warning" is an alert or notification issued to alert the user.

[1086] "Transcription" is the process of converting audio into text data.

[1087] An "illustration" is a diagram or picture that visually represents information.

[1088] A "sensor" is a device that senses physical data (e.g., motion or sound) and converts that data into an electronic signal.

[1089] "Analysis" is the process of examining collected data in detail to understand its meaning and patterns.

[1090] "Voice instructions" are instructions or commands given to the user using voice.

[1091] The system for realizing the present invention is configured using the following hardware and software.

[1092] Hardware

[1093] 1. Smartphone (e.g. iPhone or Android device)

[1094] 2. Smart glasses (e.g. Microsoft HoloLens)

[1095] 3. Sensors (accelerometer, gyro sensor, camera, etc.)

[1096] software

[1097] 1. Speech recognition library (e.g. Google Speech-to-Text API)

[1098] 2. Machine learning frameworks (e.g., TensorFlow)

[1099] 3. Real-time motion analysis library (e.g., OpenCV)

[1100] System configuration and processing

[1101] 1. Collecting user hearing and movement data:

[1102] The user wears the hearing aid, and the system collects real-time hearing and motion data through motion sensors and microphones, for example, continuously monitoring the worker's movements and surrounding sounds while working in a factory.

[1103] 2. Data transmission and analysis on the server:

[1104] The collected data is sent via smartphones or smart glasses to a server, which receives the collected data and analyzes it using a generative AI model (e.g., GPT-4).

[1105] 3. Recommend and apply optimal settings:

[1106] The server analyzes the data and then suggests optimal volume and filter settings. This makes it easier to hear work instructions, even in noisy factory environments. The analysis results are sent to the hearing aid device, which automatically adjusts the settings.

[1107] 4. Fall risk assessment and alert notification:

[1108] The system evaluates the risk of falls based on the worker's movement data, and if a high risk is detected, a warning is issued via smartphone or smart glasses. If a fall actually occurs, a notification is sent immediately to the manager.

[1109] 5. Transcription and visualization of audio instructions:

[1110] Work instructions and meeting content are transcribed in real time, and a generative AI model extracts key points and converts them into easy-to-understand illustrations. The converted illustrations and transcription data are displayed on the user's smartphone or smart glasses.

[1111] Specific examples

[1112] When workers performing maintenance on equipment in a factory have difficulty hearing instructions due to excessive noise, the system optimizes voice instructions and continuously monitors the risk of falls to enhance safety. Meetings are also transcribed in real time, and a generative AI model summarizes and displays relevant illustrations to enhance worker understanding.

[1113] Prompt Sentence Examples

[1114] Please convert the following audio data into text and summarize it. When doing so, please extract key points so that the summary can be converted into an easy-to-understand illustration.

[1115] Audio data: {Audio data}

[1116] This configuration makes it possible to optimize voice instructions to suit the work environment, improve worker safety, and visualize information, all of which are the goals of the invention.

[1117] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1118] Step 1:

[1119] The user puts on the hearing aid and launches the application installed on the device. The device begins collecting hearing data and movement data. Specifically, it uses the microphone and sensors built into the hearing aid to monitor the user's surrounding sound environment and movement in real time. Input: Environmental sound, movement data. Output: Collected raw data.

[1120] Step 2:

[1121] The device sends the collected auditory and movement data to a server. The data is transmitted over a network using a secure communication protocol (e.g., HTTPS). Input: Collected raw data. Output: Data transmitted to the server.

[1122] Step 3:

[1123] The server processes the received data and analyzes it using a generative AI model. It analyzes the data in real time using a speech recognition library (e.g., Google Speech-to-Text API) and a machine learning framework (e.g., TensorFlow). Input: Transmitted data. Output: Analysis results (optimal hearing aid settings, fall risk assessment, transcription results, etc.).

[1124] Step 4:

[1125] The server then proposes optimal hearing aid settings based on the analysis results. Specifically, the generative AI model calculates volume and filter settings appropriate for the sound environment and sends the settings information to the device. Input: Analysis results. Output: Hearing aid setting information.

[1126] Step 5:

[1127] The device uses the received hearing aid setting information to automatically adjust the hearing aid settings. The processor inside the hearing aid receives this and adjusts the volume and filter settings in real time. Input: Hearing aid setting information. Output: Automatically adjusted hearing aid settings.

[1128] Step 6:

[1129] The device also continues to analyze movement data and issues an alert to the user if it determines there is a high risk of falling. Specifically, it notifies the user by sound warning or vibration. In addition, if a fall actually occurs, it immediately sends a notification to emergency contacts. Input: Movement data analyzed in real time. Output: Warning alert, emergency notification.

[1130] Step 7:

[1131] When a user is participating in a conversation or meeting, the device converts the collected voice data into text and sends the text data to the server. Input: Voice data. Output: Text data.

[1132] Step 8:

[1133] The server receives text data and uses a generative AI model to extract key points and convert them into easy-to-understand illustrations. Input: Text data. Output: Key points, illustrations.

[1134] Step 9:

[1135] The device visually displays the illustrations and transcription data received from the server to the user. This information is displayed on the display of a smartphone or smart glasses to help the user understand. Input: Illustrations, transcription data. Output: Visually displayed information.

[1136] Through these processing steps, the system supports the user's hearing and movements in real time, providing optimal hearing aid settings, improved safety, and information visualization.

[1137] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1138] The present invention is an innovative hearing aid system that utilizes generative AI models and incorporates user emotion recognition functionality. Specific embodiments of each function are described in detail below.

[1139] Optimizing hearing aid settings

[1140] When a user puts on their hearing aid and launches the settings app, the device automatically optimizes the hearing aid settings based on their hearing environment. When the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated on the server analyzes the data and suggests optimal hearing aid settings. The analysis results are sent back to the device, which automatically adjusts the hearing aid settings.

[1141] For example, if a user points out that they have difficulty hearing voices in a movie theater, the hearing aid device analyzes the sound environment and the AI ​​model suggests optimal volume and filter settings. The hearing aid device then applies the settings and the user confirms the sound quality.

[1142] Fall risk detection and prevention

[1143] This includes a function to detect and prevent the risk of falls by collecting user movement data in real time and sending it to a server. Cameras and sensors built into the hearing aid device monitor the user's movements and collect movement data. This data is sent to the server, where it is analyzed by a generative AI model and used to assess the risk of falls. If a high risk of falling is detected, the server sends an alert to the device, which then issues an audio or vibration alert to the user. Additionally, if the user actually falls, the hearing aid device automatically sends an alert to emergency contacts.

[1144] For example, if a user is about to lose balance while descending stairs, the hearing aid device will immediately sound an alarm to alert the user, and if they still fall, an alert will be sent to emergency contacts.

[1145] Transcription and illustration of conversation content

[1146] The system collects conversational audio in real time, transcribes it, and converts it into illustrations to help users understand the content of the conversation. When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is then converted into illustrations, which the device displays on the user's device screen.

[1147] For example, if a user is having difficulty understanding a conversation while dining with a friend at a restaurant, the hearing aid will transcribe the conversation and display the analysis results along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," an illustration and key words related to the movie will be displayed on the screen.

[1148] Summarizing and illustrating online meetings

[1149] The system has the ability to collect audio from online meetings in real time, summarize it, and then illustrate and display it. Users participate in online meetings and connect their hearing aids to their PCs or smartphones to collect audio data from the meeting. The audio data is then converted into text and sent to a server. A generative AI model analyzes and summarizes the meeting content, and the summary is then converted into an illustration that is easier to understand visually. The results are then received by the terminal and displayed on the user's device.

[1150] For example, when a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, and the summary and illustrations are analyzed by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content and an illustration of the new product are displayed.

[1151] Incorporating an emotion engine

[1152] The system also incorporates an emotion engine that recognizes emotions from the user's facial expressions and voice, recognizing the user's emotional state in real time and adjusting other system functions accordingly.

[1153] For example, if a user expresses anxiety or irritation during a conversation, the emotion engine will recognize this and adjust the hearing aid settings appropriately. It will also change the illustrations and transcriptions provided based on the emotion to provide information that is best suited to the user's emotional state.

[1154] For example, if a user is feeling stressed during a business meeting, the emotion engine will detect that stress, summarize the conversation briefly, and display an illustration to help reduce stress. The tone and volume of the audio will also be adjusted, allowing the user to participate in the meeting in a more relaxed state.

[1155] The processing flow will be explained below.

[1156] Optimizing hearing aid settings

[1157] Processing Steps

[1158] Step 1:

[1159] The user puts on the hearing aid and launches the settings app on their smartphone or the hearing aid.

[1160] Step 2:

[1161] The user presses the "Optimize Settings" button in the app.

[1162] Step 3:

[1163] The device (hearing aid or smartphone) records the surrounding sound environment for a few seconds and collects auditory data such as volume and frequency spectrum.

[1164] Step 4:

[1165] The terminal transmits the collected auditory data to a server.

[1166] Step 5:

[1167] Based on the data received by the server, a generative AI model is run, which analyzes the auditory data and suggests optimal settings.

[1168] Step 6:

[1169] The server returns the configuration data including the analysis results to the terminal.

[1170] Step 7:

[1171] The device applies the received configuration data and automatically adjusts the audio settings of the hearing aid.

[1172] Step 8:

[1173] The user will see a "Setup complete" notification and be asked to review their listening progress and provide feedback.

[1174] Step 9:

[1175] If the user instructs fine adjustment as needed, the setting data is sent to the server again and adjustments are made using the same process.

[1176] Fall risk detection and prevention

[1177] Processing Steps

[1178] Step 1:

[1179] The user wears the hearing aid and goes about their daily activities.

[1180] Step 2:

[1181] The device (camera and sensors built into the hearing aid) collects the user's behavior data in real time.

[1182] Step 3:

[1183] The operating data collected by the device is temporarily stored and an initial analysis is performed using an anomaly detection algorithm.

[1184] Step 4:

[1185] The device sends the analysis results to the server and requests a detailed analysis.

[1186] Step 5:

[1187] Based on the data received by the server, a generative AI model is run to assess the risk of falling.

[1188] Step 6:

[1189] The server returns the results of the risk assessment to the terminal.

[1190] Step 7:

[1191] If the device is judged to be at high risk of falling, it will issue an alert to the user via voice or vibration to warn them.

[1192] Step 8:

[1193] If the user falls, the device will detect the fall and automatically send a text message or phone alert to pre-defined emergency contacts.

[1194] Transcription and illustration of conversation content

[1195] Processing Steps

[1196] Step 1:

[1197] The user puts on the hearing aid and begins a conversation.

[1198] Step 2:

[1199] The device (hearing aid) collects conversational audio in real time.

[1200] Step 3:

[1201] The device temporarily stores the voice data and performs initial voice recognition.

[1202] Step 4:

[1203] The terminal transmits the converted text data to the server.

[1204] Step 5:

[1205] Based on the text data received by the server, a generative AI model is run to analyze keywords and important points.

[1206] Step 6:

[1207] Based on the analysis results, the server converts the content into easy-to-understand illustrations.

[1208] Step 7:

[1209] The server sends the generated illustrations and transcription data back to the device.

[1210] Step 8:

[1211] The device (hearing aid or connected smartphone) displays the transcription results and illustrations on the user's screen.

[1212] Summarizing and illustrating online meetings

[1213] Processing Steps

[1214] Step 1:

[1215] A user joins an online meeting.

[1216] Step 2:

[1217] The user connects the conferencing app on their PC or smartphone to their hearing aid.

[1218] Step 3:

[1219] The device (hearing aid or PC / smartphone) collects the conference audio in real time.

[1220] Step 4:

[1221] The device transcribes the audio data and sends the generated text data to the server.

[1222] Step 5:

[1223] Based on the text data received by the server, a generative AI model is run to generate a summary of the meeting contents.

[1224] Step 6:

[1225] Based on the summary results, the server illustrates important points and keywords.

[1226] Step 7:

[1227] The server sends the summary and illustrations back to the device.

[1228] Step 8:

[1229] The device (hearing aid or connected smartphone) displays a summary and illustrations on the user's screen, allowing the user to efficiently understand the meeting content.

[1230] Incorporating an emotion engine

[1231] Processing Steps

[1232] Step 1:

[1233] The user wears the hearing aid and participates in daily activities, conversations, and online meetings.

[1234] Step 2:

[1235] The camera and microphone on the device (hearing aid or smartphone) collect the user's facial expressions and voice in real time.

[1236] Step 3:

[1237] The device uses the data collected to run an emotion engine and recognize the user's emotions.

[1238] Step 4:

[1239] The device transmits the recognized emotion data to the server.

[1240] Step 5:

[1241] The server runs a generative AI model based on the emotional data and analyzes the optimal hearing aid settings and information delivery methods.

[1242] Step 6:

[1243] The server returns the analysis results to the device.

[1244] Step 7:

[1245] The device adjusts the hearing aid settings and provides different illustrations and transcriptions depending on the user's emotional state.

[1246] Step 8:

[1247] Users can receive hearing aid settings and information that are optimized for their emotional state.

[1248] Example 2

[1249] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1250] While modern hearing aids offer a wide range of functions, they face challenges in adapting to individual user needs and circumstances. In particular, they lack the ability to automatically adjust settings in response to environmental changes, or to recognize the user's emotions and behavior in real time and respond appropriately. Furthermore, technologies for making online meetings and everyday conversations easier to understand are limited. This makes it difficult to provide effective support for the hearing impaired and elderly to live comfortably.

[1251] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting auditory data of the user, means for transmitting the collected auditory data to the server, means for analyzing the auditory data using an AI model generated by the server and proposing optimal hearing aid device settings, means for automatically adjusting hearing aid device settings based on the analysis results, means for collecting user movement data in real time, means for transmitting the collected movement data to the server, means for assessing a fall risk using an AI model generated by the server, means for alerting the user when a high risk of falling is detected, means for sending an alert to an emergency contact when a fall occurs, means for collecting conversation voice in real time, means for converting the collected voice data into text data, means for transmitting the text data to the server, and means for converting the collected voice data into text data using an AI model generated by the server. The system includes means for analyzing audio data from online meetings and extracting important points, means for converting content including important points into easy-to-understand images, means for displaying the converted images and transcription data to the user, means for collecting the user's emotional state, means for transmitting the collected emotional data to a server, means for analyzing the emotional data using an AI model generated by the server and adjusting other system functions, means for providing information according to the emotions, means for collecting audio data from online meetings in real time, means for converting the collected audio data into text data, means for transmitting the text data to a server, means for analyzing the text data using an AI model generated by the server and summarizing it, means for converting the summarized content into easy-to-understand images, and means for displaying the converted images and transcription data to the user. This enables automatic adaptation to the environment and situation, advance detection and prevention of fall risks, support for understanding conversations and online meeting content, and appropriate responses based on the user's emotions.

[1252] "Hearing data" refers to data that includes various parameters related to hearing, such as volume and frequency spectrum.

[1253] A "hearing assistive device" is a device worn on the ear that amplifies or converts audio signals to assist the user's hearing.

[1254] "Motion Data" is data that includes a user's physical movements and location information and is collected by sensors.

[1255] The "fall risk" is an evaluation value of the possibility that the user will fall, and is calculated based on the motion data.

[1256] "Audio Data" refers to audio signals collected by a microphone and converted into digital form.

[1257] "Text data" is data that has been converted from voice data into characters or words and is expressed as a sentence.

[1258] "Important points" are information or keywords that are particularly important to the user and are extracted from the text data.

[1259] An "image" is a figure or picture that is generated to make information easier to understand visually.

[1260] "Emotion data" is data that indicates the emotional state of the user that can be inferred from facial expressions, tone of voice, and the like.

[1261] An "online conference" is a conference held between remote locations in real time via the Internet.

[1262] A "summary" is a concise summary of the essence or important points extracted from text or audio data.

[1263] The present invention is a hearing aid system that utilizes generative AI models and incorporates user emotion recognition. This system aims to solve the challenges faced by modern hearing aids. Specific implementations of each function are described in detail below.

[1264] Hearing data collection and setting optimization

[1265] When a user puts on their hearing aid and launches the settings app, the device records the surrounding sound environment for a few seconds, collecting hearing data such as volume and frequency spectrum. This data is sent wirelessly (via Wi-Fi or mobile data) to a server. The server analyzes the data using a generative AI model and recommends optimal hearing aid settings. The analysis results are sent back to the device, which then automatically adjusts the hearing aid settings.

[1266] Example: If a user points out that they have difficulty hearing voices in a movie theater, the device analyzes the sound environment in the theater, and the AI ​​model suggests optimal volume and filter settings. The device then applies these suggestions to the hearing aid, and the user can check the sound quality.

[1267] Fall risk detection and prevention

[1268] The user's movement data is collected in real time by cameras and sensors built into the hearing aid device. This data is sent to a server, where a generative AI model analyzes the movement data and assesses the risk of falling. If a high risk of falling is detected, the server sends an alert to the device, which then issues an audio or vibration alert to the user. If a fall actually occurs, the hearing aid device automatically sends an alert to emergency contacts.

[1269] Example: If a user loses balance while descending stairs, the device will immediately sound an alarm to alert the user, and if the user still falls, an alert will be automatically sent to emergency contacts.

[1270] Transcription and illustration of conversation content

[1271] When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio in real time and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is then converted into an illustration, which the device displays on the user's device screen.

[1272] Example: If a user is having difficulty understanding the conversation while dining with friends at a restaurant, the device will transcribe the conversation and display the analysis results along with illustrations on the smartphone.

[1273] Example prompt: "I saw a movie yesterday," followed by "Show me an illustration and key words related to the movie."

[1274] Summarizing and illustrating online meetings

[1275] Users participate in online meetings and connect their hearing aids to their PCs or smartphones to collect audio data from the meeting. This data is converted into text and sent to a server. A generative AI model analyzes the meeting content and converts the summary into an illustration. The result is sent to the user's device and displayed.

[1276] Example: When a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, and the summary and illustrations are analyzed by the server and displayed on the smartphone.

[1277] Example prompt: "Show me a summary and related illustration of your new product launch plan."

[1278] Built-in emotion recognition engine

[1279] The system incorporates an emotion engine that recognizes the user's emotional state in real time from facial expressions and vocal tone. This emotion engine adjusts the system settings appropriately based on the user's emotions. The emotion engine provides information and adjusts settings appropriate to the user's emotions based on other system functions.

[1280] Example: If a user is feeling stressed during a business meeting, the emotion engine will recognize the stress, summarize the conversation, and display an illustration to help the user reduce stress. It will also adjust the tone and volume of the audio.

[1281] Example prompt: "The user is feeling stressed. Please provide information and images that will encourage relaxation."

[1282] The system of the present invention has the above-mentioned functions and is intended to optimize the user's auditory experience in various situations while improving safety and information provision.

[1283] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1284] Optimizing hearing aid settings

[1285] Step 1:

[1286] The user puts on the hearing aid and launches the settings app on their smartphone.

[1287] Input: The user puts on the hearing aid and touches it to launch the app.

[1288] Output: The app's home screen is displayed.

[1289] Specific behavior: The user selects and taps the Settings app icon on the smartphone's home screen.

[1290] Step 2:

[1291] When the user presses the "Optimize Settings" button in the app, the device begins recording the surrounding sound environment.

[1292] Input: User taps the "Optimize Settings" button.

[1293] Output: Sound environment data (volume, frequency spectrum) is collected.

[1294] What it does: The device's microphone is activated and collects the sound environment for a few seconds.

[1295] Step 3:

[1296] The device transmits the collected sound environment data to the server.

[1297] Input: Collected sound environment data.

[1298] Output: Sound environment data sent to the server.

[1299] What happens: Your device uploads data using Wi-Fi or mobile data.

[1300] Step 4:

[1301] The server analyzes the data using a generative AI model and suggests optimal settings for the hearing aid.

[1302] Input: Sound environment data sent to the server.

[1303] Output: Optimal setting parameters (volume settings and filter settings).

[1304] What it does: The server runs a Python script, analyzes data with an AI model, and calculates configuration parameters.

[1305] Step 5:

[1306] The server sends the analysis results back to the device, which then automatically adjusts the hearing aid settings.

[1307] Input: The optimal configuration parameters sent by the server.

[1308] Output: The applied hearing aid settings.

[1309] Specific operation: The device applies the received configuration parameters to the hearing aid system.

[1310] Fall risk detection and prevention

[1311] Step 1:

[1312] Cameras and sensors built into the device collect user behavior data in real time.

[1313] Input: User movement data from cameras and sensors.

[1314] Output: Collected behavioral data.

[1315] How it works: The hearing aid's accelerometer and gyroscope sensors continuously monitor and record the user's movements.

[1316] Step 2:

[1317] This motion data is sent to the server.

[1318] Input: Collected behavioral data.

[1319] Output: The operation data sent to the server.

[1320] What happens: Your device compresses the data and uploads it to the server over a secure connection.

[1321] Step 3:

[1322] The server uses the generated AI model to analyze movement data and assess the risk of falling.

[1323] Input: The operational data sent to the server.

[1324] Output: Fall risk assessment score.

[1325] Specific behavior: The server identifies patterns and anomalous behavior and uses a scoring algorithm to assess risk.

[1326] Step 4:

[1327] If a high risk of falling is detected, the server sends an alert to the device, which then warns the user with sound or vibration.

[1328] Input: Fall risk assessment score.

[1329] Output: Warning alert (audio and vibration) by the device.

[1330] What happens: The device plays a specific audio file and vibrates.

[1331] Step 5:

[1332] If a fall occurs, the hearing aid device will send an alert to emergency contacts.

[1333] Input: Actual fall data.

[1334] Output: Alert sent to emergency contacts.

[1335] What happens: Your device will send an SMS or automated call to your emergency contacts.

[1336] Transcription and illustration of conversation content

[1337] Step 1:

[1338] When a user starts a conversation, the terminal collects the conversation voice in real time.

[1339] Input: Audio data from a conversation.

[1340] Output: Collected audio data.

[1341] How it works: The device's microphone records the conversation and processes the data in real time.

[1342] Step 2:

[1343] The device transcribes the collected voice data and converts it into text data.

[1344] Input: Collected audio data.

[1345] Output: Transcribed text data.

[1346] Specific operation: The device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the voice data into text.

[1347] Step 3:

[1348] The converted text data is sent to the server.

[1349] Input: Transcribed text data.

[1350] Output: The text data sent to the server.

[1351] What happens: Your device uploads data to the server using Wi-Fi or mobile data.

[1352] Step 4:

[1353] The server's generative AI model analyzes the text data and extracts key points.

[1354] Input: Text data sent to the server.

[1355] Output: Extracted key points.

[1356] What it does: The server uses NLP (Natural Language Processing) tools to identify and summarize the main points of the text.

[1357] Step 5:

[1358] The extracted content is converted into an illustration, which the terminal displays on the user's device screen.

[1359] Input: Extracted key points.

[1360] Output: The generated illustration and the display data that contains it.

[1361] Specific operation: The server sends the generated illustration to the device, which then displays the illustration on the user's smartphone or tablet.

[1362] Summarizing and illustrating online meetings

[1363] Step 1:

[1364] The user joins an online meeting and connects their hearing aid to their PC or smartphone.

[1365] Input: Audio data from the conference.

[1366] Output: Collected audio data.

[1367] How it works: The microphone on your PC or smartphone picks up the conference audio and transmits the audio data to your hearing aids in real time.

[1368] Step 2:

[1369] The hearing aid converts the audio data of the meeting into text and sends it to the server.

[1370] Input: Collected audio data.

[1371] Output: Transcribed text data.

[1372] What it does: The hearing aid uses a speech recognition API to convert speech into text and sends that data to the server.

[1373] Step 3:

[1374] The server uses a generative AI model to generate the summary.

[1375] Input: The text data sent.

[1376] Output: Summarized text data.

[1377] What it does: The server uses NLP tools to analyze the meeting content and summarize the key points.

[1378] Step 4:

[1379] The summary results are converted into illustrations, which the terminal displays on the user's device screen.

[1380] Input: Summarized text data.

[1381] Output: The generated illustration and the display data that contains it.

[1382] Specific operation: The server sends the generated illustration to the device, which then displays the illustration on the user's smartphone or tablet.

[1383] Built-in emotion recognition engine

[1384] Step 1:

[1385] The device collects the user's facial expressions and voice tone and analyzes the emotional data.

[1386] Input: User's facial expression data, voice tone data.

[1387] Output: Collected emotion data.

[1388] How it works: The device's camera and microphone capture the user's facial expressions and tone of voice, which are then analyzed by emotion recognition software.

[1389] Step 2:

[1390] The collected emotion data is sent to a server.

[1391] Input: Data from emotion recognition.

[1392] Output: Emotion data sent to the server.

[1393] Specific operation: The device uploads emotion data to the server using Wi-Fi or mobile data.

[1394] Step 3:

[1395] The server uses a generative AI model to analyze the emotional data and adjust the system settings.

[1396] Input: The submitted emotion data.

[1397] Output: Adjusted system configuration parameters.

[1398] How it works: The server analyzes emotional data using an AI model and optimally sets the tone of voice and the way information is presented.

[1399] Step 4:

[1400] Information is provided according to emotions, and the device adjusts the screen display and audio appropriately.

[1401] Input: Adjusted system configuration parameters.

[1402] Output: Adjusted informational and audio settings.

[1403] Specific operation: The device provides information according to the user's emotions, displays information and illustrations to encourage relaxation, and adjusts the tone of the voice.

[1404] (Application example 2)

[1405] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1406] Conventional hearing aid systems often only provide simple hearing aid functions related to the user's hearing, making them inappropriate for specific environments such as factories. They also lack features that reduce the risk of falls in work environments and make it easier to understand the work being done. The present invention aims to solve these problems and provide an advanced hearing aid system that can be used in environments such as factories.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the user's hearing data, means for transmitting the collected hearing data to the server, means for analyzing the hearing data using an AI model generated by the server and proposing optimal hearing aid settings, means for automatically adjusting the hearing aid settings based on the analysis results, and means for recording and analyzing the sound environment in a factory environment and proposing sound setting adjustments. This makes it possible to provide an optimal hearing environment even in a factory and improve work efficiency.

[1408] "Auditory data" refers to audio information related to the user's hearing, and specifically refers to collected information on sound waves such as surrounding environmental sounds and human voices.

[1409] "Server" refers to the central control unit for analyzing collected data and running generative AI models.

[1410] A "generative AI model" is a data analysis model that uses artificial intelligence algorithms executed on a server to evaluate and analyze the user's auditory and behavioral data.

[1411] "Analysis results" refers to the results of data analyzed by the generative AI model, including suggestions for hearing aid settings and fall risk assessment results.

[1412] "Hearing aid settings" refers to the operating parameters of a hearing aid to assist the user's hearing, such as adjusting the volume or frequency spectrum.

[1413] "Motion data" refers to information about a user's physical movements and posture, and is collected using sensors and cameras.

[1414] The "fall risk" refers to the evaluation result of the possibility that the user will fall, and is calculated based on the motion data.

[1415] "Emergency contacts" refers to contacts to be notified in the event of an emergency such as a fall, such as the phone numbers of medical personnel or family members.

[1416] "Conversational voice" refers to voice information in a conversation between a user and another person, and is collected using a microphone or the like.

[1417] "Text data" refers to information obtained by converting voice data into text, such as transcribed conversation content.

[1418] "Illustration" refers to a picture or diagram drawn to visually show content, including text data.

[1419] The present invention is an innovative factory work environment optimization system that utilizes a generative AI model and incorporates a user emotion recognition function. Specific embodiments for carrying out the present invention will be described in detail below.

[1420] 1. System Configuration

[1421] The system mainly consists of the following components:

[1422] Audio collection device (microphone, etc.)

[1423] Motion collection devices (cameras, sensors, etc.)

[1424] server

[1425] User device (smartphone, tablet, etc.)

[1426] 2. Program processing content

[1427] The system performs processing using the following hardware and software.

[1428] Recording and analyzing sound environments

[1429] Hardware used: Microphone

[1430] Software used: SoundAnalysis

[1431] The user device collects the sound environment in the factory using a microphone and transmits the data in real time to a server, where a generative AI model analyzes the audio data and suggests optimal volume and frequency spectrum settings.

[1432] Fall risk detection

[1433] Hardware used: Camera, sensor

[1434] Software used: Motion analysis algorithm

[1435] Cameras and sensors monitor the user's movements in real time and send the collected data to a server. The generative AI model analyzes this movement data and assesses the risk of falling. If a high risk is detected, the device will issue an alert to the user.

[1436] Summary and illustration of work

[1437] Hardware used: Microphone

[1438] Software used: SpeechToText, TextToIllustration

[1439] A microphone collects the user's conversations while they are working and sends them to the server. The generative AI model converts the voice data into text and extracts key points. The results are converted into illustrations and displayed on the user's device.

[1440] Improving work efficiency through emotion recognition

[1441] Hardware used: Camera, microphone

[1442] Software used: EmotionRecognition

[1443] The system uses cameras and microphones to collect the user's facial expressions and voice, which are then sent to a server. The generative AI model analyzes this data to recognize the user's emotional state. Based on the recognized emotions, the system automatically adjusts its work settings to reduce stress and fatigue.

[1444] 3. Specific Examples

[1445] Optimizing the sound environment

[1446] For example, if a machine in a factory makes an abnormal sound, a microphone picks up the sound and sends it to a server. The generative AI model analyzes the audio data, determines that the sound is a motor malfunction, and suggests optimal volume and filter settings. The user's device then applies the settings and checks the quality of the sound.

[1447] Fall risk detection

[1448] If the user is about to lose balance while working, the sensor detects abnormal movements and immediately issues an alert to the user, and if the fall still occurs, an alert is automatically sent to emergency contacts.

[1449] Summary of work

[1450] When a user starts talking, a microphone collects the audio and sends it to a server. The generative AI model converts the conversation into text, extracts key points, and converts them into illustrations. The illustrations and text are then displayed on the user's device, making it easier for the user to understand the task at hand.

[1451] 4. Example prompt (text format)

[1452] "Analyze the sound data below and suggest optimal earplug settings. If there is additional anxiety or irritation, generate and display appropriate illustrations."

[1453] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1454] Step 1:

[1455] The user terminal collects the sound environment in the factory using a microphone.

[1456] Input: Environmental sounds inside the factory

[1457] Output: Collected audio data

[1458] How it works: Microphones record the sounds of machinery and alarms generated within the factory in real time and store the data in digital format.

[1459] Step 2:

[1460] The collected voice data is sent to a server.

[1461] Input: Collected audio data

[1462] Output: Audio data sent to the server

[1463] Specific operation: The user device uploads the saved voice data to the server using Wi-Fi or wired communication.

[1464] Step 3:

[1465] The server uses a generative AI model to analyze the audio data and suggest optimal volume and frequency spectrum settings.

[1466] Input: Audio data sent to the server

[1467] Output: Optimal volume and frequency spectrum settings

[1468] How it works: A generative AI model on a server analyzes audio data and uses detectors for abnormal sounds and specific frequencies to calculate optimal acoustic settings.

[1469] Step 4:

[1470] The server automatically adjusts the hearing aid settings based on the analysis results.

[1471] Input: Optimal volume and frequency spectrum settings

[1472] Output: Adjusted hearing aid settings

[1473] Specific operation: The server sends adjusted setting information to the user device, and the user device updates the hearing aid settings in real time.

[1474] Step 5:

[1475] Cameras and sensors collect user behavior data in real time.

[1476] Input: User action

[1477] Output: Collected behavioral data

[1478] Specific movements: Cameras and sensors capture the user's movements, such as gait and posture, and store the data in digital form.

[1479] Step 6:

[1480] The collected operation data is sent to a server.

[1481] Input: Collected behavioral data

[1482] Output: Operational data sent to the server

[1483] Specific operation: The user terminal uploads operation data to the server using the communication infrastructure.

[1484] Step 7:

[1485] The server uses a generative AI model to analyze movement data and assess fall risk.

[1486] Input: Operational data sent to the server

[1487] Output: Fall risk assessment results

[1488] Specific movements: A generative AI model analyzes movement data and applies algorithms to detect abnormal movements and assess fall risk.

[1489] Step 8:

[1490] If a high risk of falling is detected, an alert will be issued to the user.

[1491] Input: Fall risk assessment results

[1492] Output: A warning alert to the user

[1493] Specific operation: The device will issue a warning sound or vibration to the user in real time to alert them.

[1494] Step 9:

[1495] A microphone collects the conversational audio and transmits it to a server.

[1496] Input: Conversational audio

[1497] Output: Conversational audio data sent to the server

[1498] What it does: The microphone records the audio of the conversation, stores it digitally, and sends it to a server.

[1499] Step 10:

[1500] The server converts the voice data into text data.

[1501] Input: Conversational audio data sent to the server

[1502] Output: Text data

[1503] What it does: It uses a generative AI model to transcribe audio data and convert it into text format.

[1504] Step 11:

[1505] The server analyzes the text data and extracts important points.

[1506] Input: Text data

[1507] Output: Key points excerpt

[1508] How it works: The generative AI model analyzes the conversation, identifying and extracting keywords and key phrases.

[1509] Step 12:

[1510] The extracted important points are converted into illustrations and displayed on the user's device.

[1511] Input: Key points excerpt

[1512] Output: Visually easy-to-understand illustrations

[1513] Specific operation: The generative AI model creates an illustration based on key points and displays it on the user's device.

[1514] Step 13:

[1515] The camera and microphone collect the user's facial expressions and voice and transmit them to the server.

[1516] Input: User's facial expression and voice data

[1517] Output: Facial expression and voice data sent to the server

[1518] How it works: The camera and microphone record the user's facial expressions and voice in real time and send them digitally to a server.

[1519] Step 14:

[1520] The server uses an emotion recognition algorithm to recognize the user's emotional state.

[1521] Input: Facial expression and voice data sent to the server

[1522] Output: User's emotional state

[1523] How it works: The generative AI model analyzes the user's facial and voice data to identify emotions.

[1524] Step 15:

[1525] The system adjusts its settings based on the user's emotional state and provides the user with information accordingly.

[1526] Input: User's emotional state

[1527] Output: Adjusted system settings and information provided

[1528] How it works: The server calculates the optimal settings based on the user's emotional state and sends that information to the user's device. The device then adjusts the system settings in real time and provides information based on the emotion.

[1529] The above are the specific processing steps.

[1530] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1531] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1532] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1533] [Third embodiment]

[1534] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1535] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1536] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1537] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1538] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1539] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1540] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1541] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1542] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1543] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1544] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1545] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1546] The present invention is an innovative hearing aid system that utilizes generative AI models and is equipped with multiple functions. Specific embodiments are described below.

[1547] Optimizing hearing aid settings

[1548] This hearing aid system has the ability to automatically optimize hearing aid settings based on the user's hearing environment by simply putting on the hearing aid and launching the settings app. When the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal hearing aid settings. The analysis results are sent back to the device, which automatically adjusts the hearing aid settings. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[1549] For example, if a user finds it difficult to hear people during a meeting, they can optimize their hearing aid settings. The hearing aid device then analyzes the sound environment of the meeting, and the AI ​​model proposes optimal volume and filter settings. The hearing aid device then applies the proposed settings, and the user can confirm the actual effect.

[1550] Fall risk detection and prevention

[1551] The system detects and prevents the risk of falls by collecting user movement data in real time and sending it to a server. Cameras and sensors built into the hearing aid device monitor the user's movements and collect movement data. This data is temporarily stored in the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends an alert to the device, which then alerts the user via audio and vibration. Additionally, if the user actually falls, the hearing aid device immediately sends an alert to emergency contacts.

[1552] For example, if a user is about to lose their balance while descending stairs, the hearing aid device will immediately sound an alarm to alert the user, and if they still fall, an alert will be sent to their family or caregiver.

[1553] Transcription and illustration of conversation content

[1554] The system collects conversational audio in real time, transcribes it, and converts it into illustrations to help users understand the content of the conversation. When a user puts on their hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[1555] For example, if a user is having difficulty understanding the conversation while dining with a friend at a restaurant, the hearing aid will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will appear on the screen.

[1556] Summarizing and illustrating online meetings

[1557] The system collects audio from online meetings in real time, converts the content into text, summarizes it, and displays it as illustrations. Users participate in online meetings, and their hearing aids connect to their PCs or smartphones to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen by the hearing aid terminal or a connected device.

[1558] For example, when a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed, along with an illustration of the new product.

[1559] The above is a detailed description of the mode for carrying out the invention.

[1560] The processing flow will be explained below.

[1561] Optimizing hearing aid settings

[1562] Processing Steps

[1563] Step 1:

[1564] The user puts on the hearing aid and launches the settings app on their smartphone or the hearing aid.

[1565] Step 2:

[1566] The user presses the "Optimize Settings" button in the app.

[1567] Step 3:

[1568] The device (hearing aid or smartphone) records the surrounding sound environment for a few seconds and collects auditory data such as volume and frequency spectrum.

[1569] Step 4:

[1570] The terminal transmits the collected auditory data to a server.

[1571] Step 5:

[1572] Based on the data received by the server, a generative AI model is run, which analyzes the auditory data and suggests optimal settings.

[1573] Step 6:

[1574] The server returns the configuration data including the analysis results to the terminal.

[1575] Step 7:

[1576] The device applies the received configuration data and automatically adjusts the audio settings of the hearing aid.

[1577] Step 8:

[1578] The user will see a "Setup complete" notification and be asked to review their listening progress and provide feedback.

[1579] Step 9:

[1580] If the user instructs fine adjustment as needed, the setting data is sent to the server again and adjustments are made using the same process.

[1581] Fall risk detection and prevention

[1582] Processing Steps

[1583] Step 1:

[1584] The user wears the hearing aid and goes about their daily activities.

[1585] Step 2:

[1586] The device (camera and sensors built into the hearing aid) collects the user's behavior data in real time.

[1587] Step 3:

[1588] The operating data collected by the device is temporarily stored and an initial analysis is performed using an anomaly detection algorithm.

[1589] Step 4:

[1590] The device sends the analysis results to the server and requests a detailed analysis.

[1591] Step 5:

[1592] Based on the data received by the server, a generative AI model is run to assess the risk of falling.

[1593] Step 6:

[1594] The server returns the results of the risk assessment to the terminal.

[1595] Step 7:

[1596] If the device is judged to be at high risk of falling, it will issue an alert to the user via voice or vibration to warn them.

[1597] Step 8:

[1598] If the user falls, the device will detect the fall and automatically send a text message or phone alert to pre-defined emergency contacts.

[1599] Transcription and illustration of conversation content

[1600] Processing Steps

[1601] Step 1:

[1602] The user puts on the hearing aid and begins a conversation.

[1603] Step 2:

[1604] The device (hearing aid) collects conversational audio in real time.

[1605] Step 3:

[1606] The device temporarily stores the voice data and performs initial voice recognition.

[1607] Step 4:

[1608] The terminal transmits the converted text data to the server.

[1609] Step 5:

[1610] Based on the text data received by the server, a generative AI model is run to analyze keywords and important points.

[1611] Step 6:

[1612] Based on the analysis results, the server converts the content into easy-to-understand illustrations.

[1613] Step 7:

[1614] The server sends the generated illustrations and transcription data back to the device.

[1615] Step 8:

[1616] The device (hearing aid or connected smartphone) displays the transcription results and illustrations on the user's screen.

[1617] Summarizing and illustrating online meetings

[1618] Processing Steps

[1619] Step 1:

[1620] A user joins an online meeting.

[1621] Step 2:

[1622] The user connects the conferencing app on their PC or smartphone to their hearing aid.

[1623] Step 3:

[1624] The device (hearing aid or PC / smartphone) collects the conference audio in real time.

[1625] Step 4:

[1626] The device transcribes the audio data and sends the generated text data to the server.

[1627] Step 5:

[1628] Based on the text data received by the server, a generative AI model is run to generate a summary of the meeting contents.

[1629] Step 6:

[1630] Based on the summary results, the server illustrates important points and keywords.

[1631] Step 7:

[1632] The server sends the summary and illustrations back to the device.

[1633] Step 8:

[1634] The device (hearing aid or connected smartphone) displays a summary and illustrations on the user's screen, allowing the user to efficiently understand the meeting content.

[1635] Example 1

[1636] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1637] Conventional hearing aid systems have difficulty in properly adjusting to the user's individual hearing environment and operating conditions. Furthermore, proper settings according to the environment are often performed manually, placing a burden on the user. Furthermore, functions for detecting the risk of falls and assisting in understanding conversation content are lacking, making them insufficient for improving the user's safety and quality of daily life. A comprehensive system that solves these problems is needed.

[1638] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1639] In this invention, the server includes means for analyzing the user's hearing data and proposing optimal hearing aid settings, means for analyzing the user's motion data and assessing the risk of falling, means for analyzing conversational audio, extracting key points and illustrating them, and means for analyzing conference audio, creating summaries and illustrating them, thereby improving the user's hearing environment and safety, and enabling them to understand the content of conversations and conferences.

[1640] "Hearing data" refers to data related to the user's hearing, such as volume and frequency spectrum obtained through a hearing aid.

[1641] "Motion data" is data that includes information about the user's body movements and posture.

[1642] A "server" is a computer system that works in conjunction with hearing aid terminals via a network to analyze and process data.

[1643] A "generative AI model" is a machine learning algorithm that analyzes collected data and generates optimal suggestions and results.

[1644] "Analysis" is the process of investigating and examining collected data in detail using specialized methods to derive useful information and conclusions.

[1645] "Hearing aid settings" are settings that adjust various parameters of the hearing aid (e.g., volume, frequency filters, etc.).

[1646] "Fall risk" is an assessment that indicates the degree of possibility that the user will fall.

[1647] An "alert" is a notification that warns or alerts the user.

[1648] "Conversational voice" refers to the voice that a user utters while communicating with others.

[1649] "Text data" refers to data obtained by converting voice data into text format.

[1650] "Important points" are parts of a conversation or meeting that are judged to be particularly important.

[1651] An "illustration" is an image that visually represents an important point or summary.

[1652] A "summary" is a concise summary of long information such as that from a meeting.

[1653] The present invention is an innovative hearing assistance system that utilizes a generative AI model and is equipped with multiple functions. Specific embodiments are described below.

[1654] Optimizing hearing aid settings

[1655] This hearing assistance system has the function of automatically optimizing settings based on the hearing environment when the user wears the hearing assistance device and launches the settings app on their smartphone. Specifically, when the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal settings. The analysis results are sent back to the device, which automatically adjusts the settings of the hearing assistance device. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[1656] For example, if a user finds it difficult to hear people during a meeting, they can optimize their settings. The device will then analyze the sound environment of the meeting, and the AI ​​model will suggest optimal volume and filter settings. The device will then apply the suggested settings, and the user can check the actual effect.

[1657] Fall risk detection and prevention

[1658] The system has the ability to detect and prevent the risk of falls by collecting user movement data in real time and sending it to a server. The device's built-in cameras and sensors monitor the user's movements and collect movement data. This data is temporarily stored on the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends a warning alert to the device, and the device alerts the user via audio and vibration. Additionally, if the user actually falls, the device immediately sends an alert to emergency contacts.

[1659] For example, if a user is about to lose balance while descending stairs, the device will immediately sound an alarm to warn the user, and if the user still falls, an alert will be sent to a family member or caregiver.

[1660] Transcription and illustration of conversation content

[1661] The system collects conversational audio in real time, transcribes it, and converts it into easy-to-understand illustrations. When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[1662] For example, if a user is having difficulty understanding the conversation while dining with friends at a restaurant, the hearing aid device will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will be displayed on the screen.

[1663] Summarizing and illustrating online meetings

[1664] The system has the ability to collect audio from online meetings in real time, convert the content into text, summarize it, and display it as illustrations. Users participate in online meetings, and the hearing aid connects with a PC or smartphone to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen via their terminal or a connected device.

[1665] For example, when a user participates in a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed on the screen, along with an illustration of the new product.

[1666] Prompt Sentence Examples

[1667] "Please suggest the best assistive hearing device settings for me during the meeting."

[1668] "Send an alert when the user loses balance while descending stairs."

[1669] "Please transcribe the conversation in the restaurant and show it in illustrations."

[1670] "Please summarize and illustrate the contents of the Zoom meeting."

[1671] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1672] Optimizing hearing aid settings

[1673] Step 1:

[1674] The user launches the Settings app on their smartphone and presses the "Optimize Settings" button.

[1675] Input: User actions

[1676] Output: Triggers configuration optimization

[1677] At this point, the setting optimization process is initiated by the user.

[1678] Step 2:

[1679] The device will use the built-in microphone to record the surrounding sound environment for a few seconds.

[1680] Input: Trigger settings optimization

[1681] Output: Sound environment data

[1682] Specifically, the device captures ambient sounds during the meeting.

[1683] Step 3:

[1684] The device extracts auditory data such as volume and frequency spectrum.

[1685] Input: Sound environment data

[1686] Output: Auditory data

[1687] The collected sound environment data is analyzed and the necessary auditory data is extracted.

[1688] Step 4:

[1689] The terminal compresses the auditory data and prepares it for transmission to the server.

[1690] Input: Auditory data

[1691] Output: Compressed data

[1692] Data such as volume and frequency spectrum is compressed and converted into a format that is sent to the server.

[1693] Step 5:

[1694] The terminal transmits the compressed data to the server.

[1695] Input: Compressed data

[1696] Output: Data on the server

[1697] Data is sent from the terminal to the server.

[1698] Step 6:

[1699] The server unpacks the received data and begins analyzing it.

[1700] Input: Compressed data

[1701] Output: Expanded data

[1702] Extract the data and prepare it for analysis.

[1703] Step 7:

[1704] An AI model generated on the server analyzes the deployment data and calculates the optimal settings.

[1705] Input: Expanded data

[1706] Output: Optimal Settings

[1707] An AI model examines the data and determines the optimal hearing aid settings for the user.

[1708] Step 8:

[1709] The server returns the analysis results to the device.

[1710] Input: Optimal Settings

[1711] Output: Configuration data on the terminal

[1712] The server communicates the optimal settings to the device.

[1713] Step 9:

[1714] The settings of the hearing aid device are automatically adjusted based on the analysis results received by the device.

[1715] Input: Setting data

[1716] Output: Calibrated hearing aids

[1717] Your device automatically changes the settings of your hearing aid.

[1718] Step 10:

[1719] The user will be notified that the setup is complete.

[1720] Input: Adjusted hearing aid

[1721] Output: Notification

[1722] The user is notified when the settings are complete, and can make fine adjustments as needed.

[1723] Fall risk detection and prevention

[1724] Step 1:

[1725] Cameras and sensors built into the device monitor and collect information about the user's behavior.

[1726] Input: User action

[1727] Output: Operation data

[1728] Obtain user behavior data in real time.

[1729] Step 2:

[1730] The operating data collected by the device is temporarily saved and an initial analysis is performed.

[1731] Input: Operation data

[1732] Output: Initial analysis results

[1733] An initial analysis is performed using an anomaly detection algorithm on the device.

[1734] Step 3:

[1735] The device sends the initial analysis results to the server.

[1736] Input: Initial analysis results

[1737] Output: Data on the server

[1738] The initial analysis results are sent to the server to prepare for detailed analysis.

[1739] Step 4:

[1740] The server expands the received data and performs a detailed fall risk assessment using a generative AI model.

[1741] Input: Initial analysis results

[1742] Output: Detailed risk assessment

[1743] The server performs detailed analysis and assesses the risk of falling.

[1744] Step 5:

[1745] If a high risk of falling is detected, the server generates an alert to warn the user.

[1746] Input: Detailed risk assessment

[1747] Output: Alert

[1748] The server generates an alert and sends it to the device.

[1749] Step 6:

[1750] The device receives an alert and sends it to the user via sound or vibration.

[1751] Input: Alert

[1752] Output: Caution

[1753] The device will notify the user of the alert by sound or vibration.

[1754] Step 7:

[1755] If the user actually falls, the device will automatically make an emergency call.

[1756] Input: Fall detection

[1757] Output: Emergency alert

[1758] It detects if the user falls and sends an alert to emergency contacts.

[1759] Transcription and illustration of conversation content

[1760] Step 1:

[1761] A user wears a hearing aid and the terminal collects conversational audio.

[1762] Input: Conversational audio

[1763] Output: Audio data

[1764] To acquire a user's conversation voice in real time.

[1765] Step 2:

[1766] The device converts the collected voice data into text.

[1767] Input: Audio data

[1768] Output: Text data

[1769] Voice recognition software is used to convert voice data into text data.

[1770] Step 3:

[1771] Sends text data to the server.

[1772] Input: Text data

[1773] Output: Data on the server

[1774] The text data is sent to the server.

[1775] Step 4:

[1776] An AI model generated on the server analyzes the text data and extracts key points.

[1777] Input: Text data

[1778] Output: Key Points

[1779] An AI model is used to analyze text data and extract key points from the conversation.

[1780] Step 5:

[1781] The server converts the extracted key points into illustrations.

[1782] Input: Key Points

[1783] Output: illustration data

[1784] Convert key points into illustrations to make them easier to understand visually.

[1785] Step 6:

[1786] The device displays the converted illustration and transcription data on the user's screen.

[1787] Input: illustration data, transcription data

[1788] Output: What the user sees on their screen

[1789] The device displays the illustration and transcription results to provide the user.

[1790] Summarizing and illustrating online meetings

[1791] Step 1:

[1792] A user participates in an online conference, and a terminal collects the conference audio.

[1793] Input: Conference audio

[1794] Output: Audio data

[1795] Collect the user's online conference audio.

[1796] Step 2:

[1797] The terminal converts the collected voice data into text data.

[1798] Input: Audio data

[1799] Output: Text data

[1800] Use speech recognition software to convert meeting audio into text data.

[1801] Step 3:

[1802] Sends text data to the server.

[1803] Input: Text data

[1804] Output: Data on the server

[1805] The text data is sent to the server.

[1806] Step 4:

[1807] The AI ​​model generated on the server analyzes the text data and creates a summary.

[1808] Input: Text data

[1809] Output: Summary data

[1810] The AI ​​model analyzes the meeting content and generates a summary.

[1811] Step 5:

[1812] The server converts the summarized content into illustrations.

[1813] Input: Summary data

[1814] Output: illustration data

[1815] Convert summarized meeting content into visually easy-to-understand illustrations.

[1816] Step 6:

[1817] The terminal displays the converted illustration and summary data on the user's screen.

[1818] Input: illustration data, summary data

[1819] Output: What the user sees on their screen

[1820] The terminal displays the illustration and summary content for the user.

[1821] (Application example 1)

[1822] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1823] Conventional hearing aid systems are limited to analyzing voice and motion data and lack optimal functionality for real-world work environments and safety. Noisy factory environments, in particular, can make it difficult to hear voice instructions and increase the risk of workers falling. A method is needed to resolve these issues and improve work efficiency and safety.

[1824] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1825] In this invention, the server

[1826] A means to analyze environmental sounds in the factory in real time and provide voice instructions to workers with optimal volume and sound quality,

[1827] A method for collecting worker movement data in real time using sensors and assessing the risk of falls.

[1828] A means to transcribe voice instructions and meeting content in real time and convert them into easy-to-understand illustrations for display,

[1829] This enables optimization of voice instructions according to the work environment, improvement of worker safety, and visualization of information.

[1830] "Auditory data" is acoustic information obtained through the user's ears, and is data that includes characteristics such as volume and frequency spectrum.

[1831] A "server" is a computer system on a network that collects, analyzes, and stores data.

[1832] An "AI model" is a model that uses machine learning algorithms to analyze data and derive optimal solutions to specific problems.

[1833] "Hearing aid settings" are various adjustment items for assisting the user's hearing, such as the volume and frequency filter settings of the hearing aid.

[1834] "Motion data" is data that includes information about the user's body movements and posture.

[1835] The "fall risk" is an evaluation value of the possibility that the user will fall, and is calculated based on the motion data.

[1836] A "warning" is an alert or notification issued to alert the user.

[1837] "Transcription" is the process of converting audio into text data.

[1838] An "illustration" is a diagram or picture that visually represents information.

[1839] A "sensor" is a device that senses physical data (e.g., motion or sound) and converts that data into an electronic signal.

[1840] "Analysis" is the process of examining collected data in detail to understand its meaning and patterns.

[1841] "Voice instructions" are instructions or commands given to the user using voice.

[1842] The system for realizing the present invention is configured using the following hardware and software.

[1843] Hardware

[1844] 1. Smartphone (e.g. iPhone or Android device)

[1845] 2. Smart glasses (e.g. Microsoft HoloLens)

[1846] 3. Sensors (accelerometer, gyro sensor, camera, etc.)

[1847] software

[1848] 1. Speech recognition library (e.g. Google Speech-to-Text API)

[1849] 2. Machine learning frameworks (e.g., TensorFlow)

[1850] 3. Real-time motion analysis library (e.g., OpenCV)

[1851] System configuration and processing

[1852] 1. Collecting user hearing and movement data:

[1853] The user wears the hearing aid, and the system collects real-time hearing and motion data through motion sensors and microphones, for example, continuously monitoring the worker's movements and surrounding sounds while working in a factory.

[1854] 2. Data transmission and analysis on the server:

[1855] The collected data is sent via smartphones or smart glasses to a server, which receives the collected data and analyzes it using a generative AI model (e.g., GPT-4).

[1856] 3. Recommend and apply optimal settings:

[1857] The server analyzes the data and then suggests optimal volume and filter settings. This makes it easier to hear work instructions, even in noisy factory environments. The analysis results are sent to the hearing aid device, which automatically adjusts the settings.

[1858] 4. Fall risk assessment and alert notification:

[1859] The system evaluates the risk of falls based on the worker's movement data, and if a high risk is detected, a warning is issued via smartphone or smart glasses. If a fall actually occurs, a notification is sent immediately to the manager.

[1860] 5. Transcription and visualization of audio instructions:

[1861] Work instructions and meeting content are transcribed in real time, and a generative AI model extracts key points and converts them into easy-to-understand illustrations. The converted illustrations and transcription data are displayed on the user's smartphone or smart glasses.

[1862] Specific examples

[1863] When workers performing maintenance on equipment in a factory have difficulty hearing instructions due to excessive noise, the system optimizes voice instructions and continuously monitors the risk of falls to enhance safety. Meetings are also transcribed in real time, and a generative AI model summarizes and displays relevant illustrations to enhance worker understanding.

[1864] Prompt Sentence Examples

[1865] Please convert the following audio data into text and summarize it. When doing so, please extract key points so that the summary can be converted into an easy-to-understand illustration.

[1866] Audio data: {Audio data}

[1867] This configuration makes it possible to optimize voice instructions to suit the work environment, improve worker safety, and visualize information, all of which are the goals of the invention.

[1868] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1869] Step 1:

[1870] The user puts on the hearing aid and launches the application installed on the device. The device begins collecting hearing data and movement data. Specifically, it uses the microphone and sensors built into the hearing aid to monitor the user's surrounding sound environment and movement in real time. Input: Environmental sound, movement data. Output: Collected raw data.

[1871] Step 2:

[1872] The device sends the collected auditory and movement data to a server. The data is transmitted over a network using a secure communication protocol (e.g., HTTPS). Input: Collected raw data. Output: Data transmitted to the server.

[1873] Step 3:

[1874] The server processes the received data and analyzes it using a generative AI model. It analyzes the data in real time using a speech recognition library (e.g., Google Speech-to-Text API) and a machine learning framework (e.g., TensorFlow). Input: Transmitted data. Output: Analysis results (optimal hearing aid settings, fall risk assessment, transcription results, etc.).

[1875] Step 4:

[1876] The server then proposes optimal hearing aid settings based on the analysis results. Specifically, the generative AI model calculates volume and filter settings appropriate for the sound environment and sends the settings information to the device. Input: Analysis results. Output: Hearing aid setting information.

[1877] Step 5:

[1878] The device uses the received hearing aid setting information to automatically adjust the hearing aid settings. The processor inside the hearing aid receives this and adjusts the volume and filter settings in real time. Input: Hearing aid setting information. Output: Automatically adjusted hearing aid settings.

[1879] Step 6:

[1880] The device also continues to analyze movement data and issues an alert to the user if it determines there is a high risk of falling. Specifically, it notifies the user by sound warning or vibration. In addition, if a fall actually occurs, it immediately sends a notification to emergency contacts. Input: Movement data analyzed in real time. Output: Warning alert, emergency notification.

[1881] Step 7:

[1882] When a user is participating in a conversation or meeting, the device converts the collected voice data into text and sends the text data to the server. Input: Voice data. Output: Text data.

[1883] Step 8:

[1884] The server receives text data and uses a generative AI model to extract key points and convert them into easy-to-understand illustrations. Input: Text data. Output: Key points, illustrations.

[1885] Step 9:

[1886] The device visually displays the illustrations and transcription data received from the server to the user. This information is displayed on the display of a smartphone or smart glasses to help the user understand. Input: Illustrations, transcription data. Output: Visually displayed information.

[1887] Through these processing steps, the system supports the user's hearing and movements in real time, providing optimal hearing aid settings, improved safety, and information visualization.

[1888] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1889] The present invention is an innovative hearing aid system that utilizes generative AI models and incorporates user emotion recognition functionality. Specific embodiments of each function are described in detail below.

[1890] Optimizing hearing aid settings

[1891] When a user puts on their hearing aid and launches the settings app, the device automatically optimizes the hearing aid settings based on their hearing environment. When the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated on the server analyzes the data and suggests optimal hearing aid settings. The analysis results are sent back to the device, which automatically adjusts the hearing aid settings.

[1892] For example, if a user points out that they have difficulty hearing voices in a movie theater, the hearing aid device analyzes the sound environment and the AI ​​model suggests optimal volume and filter settings. The hearing aid device then applies the settings and the user confirms the sound quality.

[1893] Fall risk detection and prevention

[1894] This includes a function to detect and prevent the risk of falls by collecting user movement data in real time and sending it to a server. Cameras and sensors built into the hearing aid device monitor the user's movements and collect movement data. This data is sent to the server, where it is analyzed by a generative AI model and used to assess the risk of falls. If a high risk of falling is detected, the server sends an alert to the device, which then issues an audio or vibration alert to the user. Additionally, if the user actually falls, the hearing aid device automatically sends an alert to emergency contacts.

[1895] For example, if a user is about to lose balance while descending stairs, the hearing aid device will immediately sound an alarm to alert the user, and if they still fall, an alert will be sent to emergency contacts.

[1896] Transcription and illustration of conversation content

[1897] The system collects conversational audio in real time, transcribes it, and converts it into illustrations to help users understand the content of the conversation. When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is then converted into illustrations, which the device displays on the user's device screen.

[1898] For example, if a user is having difficulty understanding a conversation while dining with a friend at a restaurant, the hearing aid will transcribe the conversation and display the analysis results along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," an illustration and key words related to the movie will be displayed on the screen.

[1899] Summarizing and illustrating online meetings

[1900] The system has the ability to collect audio from online meetings in real time, summarize it, and then illustrate and display it. Users participate in online meetings and connect their hearing aids to their PCs or smartphones to collect audio data from the meeting. The audio data is then converted into text and sent to a server. A generative AI model analyzes and summarizes the meeting content, and the summary is then converted into an illustration that is easier to understand visually. The results are then received by the terminal and displayed on the user's device.

[1901] For example, when a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, and the summary and illustrations are analyzed by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content and an illustration of the new product are displayed.

[1902] Incorporating an emotion engine

[1903] The system also incorporates an emotion engine that recognizes emotions from the user's facial expressions and voice, recognizing the user's emotional state in real time and adjusting other system functions accordingly.

[1904] For example, if a user expresses anxiety or irritation during a conversation, the emotion engine will recognize this and adjust the hearing aid settings appropriately. It will also change the illustrations and transcriptions provided based on the emotion to provide information that is best suited to the user's emotional state.

[1905] For example, if a user is feeling stressed during a business meeting, the emotion engine will detect that stress, summarize the conversation briefly, and display an illustration to help reduce stress. The tone and volume of the audio will also be adjusted, allowing the user to participate in the meeting in a more relaxed state.

[1906] The processing flow will be explained below.

[1907] Optimizing hearing aid settings

[1908] Processing Steps

[1909] Step 1:

[1910] The user puts on the hearing aid and launches the settings app on their smartphone or the hearing aid.

[1911] Step 2:

[1912] The user presses the "Optimize Settings" button in the app.

[1913] Step 3:

[1914] The device (hearing aid or smartphone) records the surrounding sound environment for a few seconds and collects auditory data such as volume and frequency spectrum.

[1915] Step 4:

[1916] The terminal transmits the collected auditory data to a server.

[1917] Step 5:

[1918] Based on the data received by the server, a generative AI model is run, which analyzes the auditory data and suggests optimal settings.

[1919] Step 6:

[1920] The server returns the configuration data including the analysis results to the terminal.

[1921] Step 7:

[1922] The device applies the received configuration data and automatically adjusts the audio settings of the hearing aid.

[1923] Step 8:

[1924] The user will see a "Setup complete" notification and be asked to review their listening progress and provide feedback.

[1925] Step 9:

[1926] If the user instructs fine adjustment as needed, the setting data is sent to the server again and adjustments are made using the same process.

[1927] Fall risk detection and prevention

[1928] Processing Steps

[1929] Step 1:

[1930] The user wears the hearing aid and goes about their daily activities.

[1931] Step 2:

[1932] The device (camera and sensors built into the hearing aid) collects the user's behavior data in real time.

[1933] Step 3:

[1934] The operating data collected by the device is temporarily stored and an initial analysis is performed using an anomaly detection algorithm.

[1935] Step 4:

[1936] The device sends the analysis results to the server and requests a detailed analysis.

[1937] Step 5:

[1938] Based on the data received by the server, a generative AI model is run to assess the risk of falling.

[1939] Step 6:

[1940] The server returns the results of the risk assessment to the terminal.

[1941] Step 7:

[1942] If the device is judged to be at high risk of falling, it will issue an alert to the user via voice or vibration to warn them.

[1943] Step 8:

[1944] If the user falls, the device will detect the fall and automatically send a text message or phone alert to pre-defined emergency contacts.

[1945] Transcription and illustration of conversation content

[1946] Processing Steps

[1947] Step 1:

[1948] The user puts on the hearing aid and begins a conversation.

[1949] Step 2:

[1950] The device (hearing aid) collects conversational audio in real time.

[1951] Step 3:

[1952] The device temporarily stores the voice data and performs initial voice recognition.

[1953] Step 4:

[1954] The terminal transmits the converted text data to the server.

[1955] Step 5:

[1956] Based on the text data received by the server, a generative AI model is run to analyze keywords and important points.

[1957] Step 6:

[1958] Based on the analysis results, the server converts the content into easy-to-understand illustrations.

[1959] Step 7:

[1960] The server sends the generated illustrations and transcription data back to the device.

[1961] Step 8:

[1962] The device (hearing aid or connected smartphone) displays the transcription results and illustrations on the user's screen.

[1963] Summarizing and illustrating online meetings

[1964] Processing Steps

[1965] Step 1:

[1966] A user joins an online meeting.

[1967] Step 2:

[1968] The user connects the conferencing app on their PC or smartphone to their hearing aid.

[1969] Step 3:

[1970] The device (hearing aid or PC / smartphone) collects the conference audio in real time.

[1971] Step 4:

[1972] The device transcribes the audio data and sends the generated text data to the server.

[1973] Step 5:

[1974] Based on the text data received by the server, a generative AI model is run to generate a summary of the meeting contents.

[1975] Step 6:

[1976] Based on the summary results, the server illustrates important points and keywords.

[1977] Step 7:

[1978] The server sends the summary and illustrations back to the device.

[1979] Step 8:

[1980] The device (hearing aid or connected smartphone) displays a summary and illustrations on the user's screen, allowing the user to efficiently understand the meeting content.

[1981] Incorporating an emotion engine

[1982] Processing Steps

[1983] Step 1:

[1984] The user wears the hearing aid and participates in daily activities, conversations, and online meetings.

[1985] Step 2:

[1986] The camera and microphone on the device (hearing aid or smartphone) collect the user's facial expressions and voice in real time.

[1987] Step 3:

[1988] The device uses the data collected to run an emotion engine and recognize the user's emotions.

[1989] Step 4:

[1990] The device transmits the recognized emotion data to the server.

[1991] Step 5:

[1992] The server runs a generative AI model based on the emotional data and analyzes the optimal hearing aid settings and information delivery methods.

[1993] Step 6:

[1994] The server returns the analysis results to the device.

[1995] Step 7:

[1996] The device adjusts the hearing aid settings and provides different illustrations and transcriptions depending on the user's emotional state.

[1997] Step 8:

[1998] Users can receive hearing aid settings and information that are optimized for their emotional state.

[1999] Example 2

[2000] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2001] While modern hearing aids offer a wide range of functions, they face challenges in adapting to individual user needs and circumstances. In particular, they lack the ability to automatically adjust settings in response to environmental changes, or to recognize the user's emotions and behavior in real time and respond appropriately. Furthermore, technologies for making online meetings and everyday conversations easier to understand are limited. This makes it difficult to provide effective support for the hearing impaired and elderly to live comfortably.

[2002] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting auditory data of the user, means for transmitting the collected auditory data to the server, means for analyzing the auditory data using an AI model generated by the server and proposing optimal hearing aid device settings, means for automatically adjusting hearing aid device settings based on the analysis results, means for collecting user movement data in real time, means for transmitting the collected movement data to the server, means for assessing a fall risk using an AI model generated by the server, means for alerting the user when a high risk of falling is detected, means for sending an alert to an emergency contact when a fall occurs, means for collecting conversation voice in real time, means for converting the collected voice data into text data, means for transmitting the text data to the server, and means for converting the collected voice data into text data using an AI model generated by the server. The system includes means for analyzing audio data from online meetings and extracting important points, means for converting content including important points into easy-to-understand images, means for displaying the converted images and transcription data to the user, means for collecting the user's emotional state, means for transmitting the collected emotional data to a server, means for analyzing the emotional data using an AI model generated by the server and adjusting other system functions, means for providing information according to the emotions, means for collecting audio data from online meetings in real time, means for converting the collected audio data into text data, means for transmitting the text data to a server, means for analyzing the text data using an AI model generated by the server and summarizing it, means for converting the summarized content into easy-to-understand images, and means for displaying the converted images and transcription data to the user. This enables automatic adaptation to the environment and situation, advance detection and prevention of fall risks, support for understanding conversations and online meeting content, and appropriate responses based on the user's emotions.

[2003] "Hearing data" refers to data that includes various parameters related to hearing, such as volume and frequency spectrum.

[2004] A "hearing assistive device" is a device worn on the ear that amplifies or converts audio signals to assist the user's hearing.

[2005] "Motion Data" is data that includes a user's physical movements and location information and is collected by sensors.

[2006] The "fall risk" is an evaluation value of the possibility that the user will fall, and is calculated based on the motion data.

[2007] "Audio Data" refers to audio signals collected by a microphone and converted into digital form.

[2008] "Text data" is data that has been converted from voice data into characters or words and is expressed as a sentence.

[2009] "Important points" are information or keywords that are particularly important to the user and are extracted from the text data.

[2010] An "image" is a figure or picture that is generated to make information easier to understand visually.

[2011] "Emotion data" is data that indicates the emotional state of the user that can be inferred from facial expressions, tone of voice, and the like.

[2012] An "online conference" is a conference held between remote locations in real time via the Internet.

[2013] A "summary" is a concise summary of the essence or important points extracted from text or audio data.

[2014] The present invention is a hearing aid system that utilizes generative AI models and incorporates user emotion recognition. This system aims to solve the challenges faced by modern hearing aids. Specific implementations of each function are described in detail below.

[2015] Hearing data collection and setting optimization

[2016] When a user puts on their hearing aid and launches the settings app, the device records the surrounding sound environment for a few seconds, collecting hearing data such as volume and frequency spectrum. This data is sent wirelessly (via Wi-Fi or mobile data) to a server. The server analyzes the data using a generative AI model and recommends optimal hearing aid settings. The analysis results are sent back to the device, which then automatically adjusts the hearing aid settings.

[2017] Example: If a user points out that they have difficulty hearing voices in a movie theater, the device analyzes the sound environment in the theater, and the AI ​​model suggests optimal volume and filter settings. The device then applies these suggestions to the hearing aid, and the user can check the sound quality.

[2018] Fall risk detection and prevention

[2019] The user's movement data is collected in real time by cameras and sensors built into the hearing aid device. This data is sent to a server, where a generative AI model analyzes the movement data and assesses the risk of falling. If a high risk of falling is detected, the server sends an alert to the device, which then issues an audio or vibration alert to the user. If a fall actually occurs, the hearing aid device automatically sends an alert to emergency contacts.

[2020] Example: If a user loses balance while descending stairs, the device will immediately sound an alarm to alert the user, and if the user still falls, an alert will be automatically sent to emergency contacts.

[2021] Transcription and illustration of conversation content

[2022] When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio in real time and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is then converted into an illustration, which the device displays on the user's device screen.

[2023] Example: If a user is having difficulty understanding the conversation while dining with friends at a restaurant, the device will transcribe the conversation and display the analysis results along with illustrations on the smartphone.

[2024] Example prompt: "I saw a movie yesterday," followed by "Show me an illustration and key words related to the movie."

[2025] Summarizing and illustrating online meetings

[2026] Users participate in online meetings and connect their hearing aids to their PCs or smartphones to collect audio data from the meeting. This data is converted into text and sent to a server. A generative AI model analyzes the meeting content and converts the summary into an illustration. The result is sent to the user's device and displayed.

[2027] Example: When a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, and the summary and illustrations are analyzed by the server and displayed on the smartphone.

[2028] Example prompt: "Show me a summary and related illustration of your new product launch plan."

[2029] Built-in emotion recognition engine

[2030] The system incorporates an emotion engine that recognizes the user's emotional state in real time from facial expressions and vocal tone. This emotion engine adjusts the system settings appropriately based on the user's emotions. The emotion engine provides information and adjusts settings appropriate to the user's emotions based on other system functions.

[2031] Example: If a user is feeling stressed during a business meeting, the emotion engine will recognize the stress, summarize the conversation, and display an illustration to help the user reduce stress. It will also adjust the tone and volume of the audio.

[2032] Example prompt: "The user is feeling stressed. Please provide information and images that will encourage relaxation."

[2033] The system of the present invention has the above-mentioned functions and is intended to optimize the user's auditory experience in various situations while improving safety and information provision.

[2034] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2035] Optimizing hearing aid settings

[2036] Step 1:

[2037] The user puts on the hearing aid and launches the settings app on their smartphone.

[2038] Input: The user puts on the hearing aid and touches it to launch the app.

[2039] Output: The app's home screen is displayed.

[2040] Specific behavior: The user selects and taps the Settings app icon on the smartphone's home screen.

[2041] Step 2:

[2042] When the user presses the "Optimize Settings" button in the app, the device begins recording the surrounding sound environment.

[2043] Input: User taps the "Optimize Settings" button.

[2044] Output: Sound environment data (volume, frequency spectrum) is collected.

[2045] What it does: The device's microphone is activated and collects the sound environment for a few seconds.

[2046] Step 3:

[2047] The device transmits the collected sound environment data to the server.

[2048] Input: Collected sound environment data.

[2049] Output: Sound environment data sent to the server.

[2050] What happens: Your device uploads data using Wi-Fi or mobile data.

[2051] Step 4:

[2052] The server analyzes the data using a generative AI model and suggests optimal settings for the hearing aid.

[2053] Input: Sound environment data sent to the server.

[2054] Output: Optimal setting parameters (volume settings and filter settings).

[2055] What it does: The server runs a Python script, analyzes data with an AI model, and calculates configuration parameters.

[2056] Step 5:

[2057] The server sends the analysis results back to the device, which then automatically adjusts the hearing aid settings.

[2058] Input: The optimal configuration parameters sent by the server.

[2059] Output: The applied hearing aid settings.

[2060] Specific operation: The device applies the received configuration parameters to the hearing aid system.

[2061] Fall risk detection and prevention

[2062] Step 1:

[2063] Cameras and sensors built into the device collect user behavior data in real time.

[2064] Input: User movement data from cameras and sensors.

[2065] Output: Collected behavioral data.

[2066] How it works: The hearing aid's accelerometer and gyroscope sensors continuously monitor and record the user's movements.

[2067] Step 2:

[2068] This motion data is sent to the server.

[2069] Input: Collected behavioral data.

[2070] Output: The operation data sent to the server.

[2071] What happens: Your device compresses the data and uploads it to the server over a secure connection.

[2072] Step 3:

[2073] The server uses the generated AI model to analyze movement data and assess the risk of falling.

[2074] Input: The operational data sent to the server.

[2075] Output: Fall risk assessment score.

[2076] Specific behavior: The server identifies patterns and anomalous behavior and uses a scoring algorithm to assess risk.

[2077] Step 4:

[2078] If a high risk of falling is detected, the server sends an alert to the device, which then warns the user with sound or vibration.

[2079] Input: Fall risk assessment score.

[2080] Output: Warning alert (audio and vibration) by the device.

[2081] What happens: The device plays a specific audio file and vibrates.

[2082] Step 5:

[2083] If a fall occurs, the hearing aid device will send an alert to emergency contacts.

[2084] Input: Actual fall data.

[2085] Output: Alert sent to emergency contacts.

[2086] What happens: Your device will send an SMS or automated call to your emergency contacts.

[2087] Transcription and illustration of conversation content

[2088] Step 1:

[2089] When a user starts a conversation, the terminal collects the conversation voice in real time.

[2090] Input: Audio data from a conversation.

[2091] Output: Collected audio data.

[2092] How it works: The device's microphone records the conversation and processes the data in real time.

[2093] Step 2:

[2094] The device transcribes the collected voice data and converts it into text data.

[2095] Input: Collected audio data.

[2096] Output: Transcribed text data.

[2097] Specific operation: The device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the voice data into text.

[2098] Step 3:

[2099] The converted text data is sent to the server.

[2100] Input: Transcribed text data.

[2101] Output: The text data sent to the server.

[2102] What happens: Your device uploads data to the server using Wi-Fi or mobile data.

[2103] Step 4:

[2104] The server's generative AI model analyzes the text data and extracts key points.

[2105] Input: Text data sent to the server.

[2106] Output: Extracted key points.

[2107] What it does: The server uses NLP (Natural Language Processing) tools to identify and summarize the main points of the text.

[2108] Step 5:

[2109] The extracted content is converted into an illustration, which the terminal displays on the user's device screen.

[2110] Input: Extracted key points.

[2111] Output: The generated illustration and the display data that contains it.

[2112] Specific operation: The server sends the generated illustration to the device, which then displays the illustration on the user's smartphone or tablet.

[2113] Summarizing and illustrating online meetings

[2114] Step 1:

[2115] The user joins an online meeting and connects their hearing aid to their PC or smartphone.

[2116] Input: Audio data from the conference.

[2117] Output: Collected audio data.

[2118] How it works: The microphone on your PC or smartphone picks up the conference audio and transmits the audio data to your hearing aids in real time.

[2119] Step 2:

[2120] The hearing aid converts the audio data of the meeting into text and sends it to the server.

[2121] Input: Collected audio data.

[2122] Output: Transcribed text data.

[2123] What it does: The hearing aid uses a speech recognition API to convert speech into text and sends that data to the server.

[2124] Step 3:

[2125] The server uses a generative AI model to generate the summary.

[2126] Input: The text data sent.

[2127] Output: Summarized text data.

[2128] What it does: The server uses NLP tools to analyze the meeting content and summarize the key points.

[2129] Step 4:

[2130] The summary results are converted into illustrations, which the terminal displays on the user's device screen.

[2131] Input: Summarized text data.

[2132] Output: The generated illustration and the display data that contains it.

[2133] Specific operation: The server sends the generated illustration to the device, which then displays the illustration on the user's smartphone or tablet.

[2134] Built-in emotion recognition engine

[2135] Step 1:

[2136] The device collects the user's facial expressions and voice tone and analyzes the emotional data.

[2137] Input: User's facial expression data, voice tone data.

[2138] Output: Collected emotion data.

[2139] How it works: The device's camera and microphone capture the user's facial expressions and tone of voice, which are then analyzed by emotion recognition software.

[2140] Step 2:

[2141] The collected emotion data is sent to a server.

[2142] Input: Data from emotion recognition.

[2143] Output: Emotion data sent to the server.

[2144] Specific operation: The device uploads emotion data to the server using Wi-Fi or mobile data.

[2145] Step 3:

[2146] The server uses a generative AI model to analyze the emotional data and adjust the system settings.

[2147] Input: The submitted emotion data.

[2148] Output: Adjusted system configuration parameters.

[2149] How it works: The server analyzes emotional data using an AI model and optimally sets the tone of voice and the way information is presented.

[2150] Step 4:

[2151] Information is provided according to emotions, and the device adjusts the screen display and audio appropriately.

[2152] Input: Adjusted system configuration parameters.

[2153] Output: Adjusted informational and audio settings.

[2154] Specific operation: The device provides information according to the user's emotions, displays information and illustrations to encourage relaxation, and adjusts the tone of the voice.

[2155] (Application example 2)

[2156] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2157] Conventional hearing aid systems often only provide simple hearing aid functions related to the user's hearing, making them inappropriate for specific environments such as factories. They also lack features that reduce the risk of falls in work environments and make it easier to understand the work being done. The present invention aims to solve these problems and provide an advanced hearing aid system that can be used in environments such as factories.

[2158] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the user's hearing data, means for transmitting the collected hearing data to the server, means for analyzing the hearing data using an AI model generated by the server and proposing optimal hearing aid settings, means for automatically adjusting the hearing aid settings based on the analysis results, and means for recording and analyzing the sound environment in a factory environment and proposing sound setting adjustments. This makes it possible to provide an optimal hearing environment even in a factory and improve work efficiency.

[2159] "Auditory data" refers to audio information related to the user's hearing, and specifically refers to collected information on sound waves such as surrounding environmental sounds and human voices.

[2160] "Server" refers to the central control unit for analyzing collected data and running generative AI models.

[2161] A "generative AI model" is a data analysis model that uses artificial intelligence algorithms executed on a server to evaluate and analyze the user's auditory and behavioral data.

[2162] "Analysis results" refers to the results of data analyzed by the generative AI model, including suggestions for hearing aid settings and fall risk assessment results.

[2163] "Hearing aid settings" refers to the operating parameters of a hearing aid to assist the user's hearing, such as adjusting the volume or frequency spectrum.

[2164] "Motion data" refers to information about a user's physical movements and posture, and is collected using sensors and cameras.

[2165] The "fall risk" refers to the evaluation result of the possibility that the user will fall, and is calculated based on the motion data.

[2166] "Emergency contacts" refers to contacts to be notified in the event of an emergency such as a fall, such as the phone numbers of medical personnel or family members.

[2167] "Conversational voice" refers to voice information in a conversation between a user and another person, and is collected using a microphone or the like.

[2168] "Text data" refers to information obtained by converting voice data into text, such as transcribed conversation content.

[2169] "Illustration" refers to a picture or diagram drawn to visually show content, including text data.

[2170] The present invention is an innovative factory work environment optimization system that utilizes a generative AI model and incorporates a user emotion recognition function. Specific embodiments for carrying out the present invention will be described in detail below.

[2171] 1. System Configuration

[2172] The system mainly consists of the following components:

[2173] Audio collection device (microphone, etc.)

[2174] Motion collection devices (cameras, sensors, etc.)

[2175] server

[2176] User device (smartphone, tablet, etc.)

[2177] 2. Program processing content

[2178] The system performs processing using the following hardware and software.

[2179] Recording and analyzing sound environments

[2180] Hardware used: Microphone

[2181] Software used: SoundAnalysis

[2182] The user device collects the sound environment in the factory using a microphone and transmits the data in real time to a server, where a generative AI model analyzes the audio data and suggests optimal volume and frequency spectrum settings.

[2183] Fall risk detection

[2184] Hardware used: Camera, sensor

[2185] Software used: Motion analysis algorithm

[2186] Cameras and sensors monitor the user's movements in real time and send the collected data to a server. The generative AI model analyzes this movement data and assesses the risk of falling. If a high risk is detected, the device will issue an alert to the user.

[2187] Summary and illustration of work

[2188] Hardware used: Microphone

[2189] Software used: SpeechToText, TextToIllustration

[2190] A microphone collects the user's conversations while they are working and sends them to the server. The generative AI model converts the voice data into text and extracts key points. The results are converted into illustrations and displayed on the user's device.

[2191] Improving work efficiency through emotion recognition

[2192] Hardware used: Camera, microphone

[2193] Software used: EmotionRecognition

[2194] The system uses cameras and microphones to collect the user's facial expressions and voice, which are then sent to a server. The generative AI model analyzes this data to recognize the user's emotional state. Based on the recognized emotions, the system automatically adjusts its work settings to reduce stress and fatigue.

[2195] 3. Specific Examples

[2196] Optimizing the sound environment

[2197] For example, if a machine in a factory makes an abnormal sound, a microphone picks up the sound and sends it to a server. The generative AI model analyzes the audio data, determines that the sound is a motor malfunction, and suggests optimal volume and filter settings. The user's device then applies the settings and checks the quality of the sound.

[2198] Fall risk detection

[2199] If the user is about to lose balance while working, the sensor detects abnormal movements and immediately issues an alert to the user, and if the fall still occurs, an alert is automatically sent to emergency contacts.

[2200] Summary of work

[2201] When a user starts talking, a microphone collects the audio and sends it to a server. The generative AI model converts the conversation into text, extracts key points, and converts them into illustrations. The illustrations and text are then displayed on the user's device, making it easier for the user to understand the task at hand.

[2202] 4. Example prompt (text format)

[2203] "Analyze the sound data below and suggest optimal earplug settings. If there is additional anxiety or irritation, generate and display appropriate illustrations."

[2204] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2205] Step 1:

[2206] The user terminal collects the sound environment in the factory using a microphone.

[2207] Input: Environmental sounds inside the factory

[2208] Output: Collected audio data

[2209] How it works: Microphones record the sounds of machinery and alarms generated within the factory in real time and store the data in digital format.

[2210] Step 2:

[2211] The collected voice data is sent to a server.

[2212] Input: Collected audio data

[2213] Output: Audio data sent to the server

[2214] Specific operation: The user device uploads the saved voice data to the server using Wi-Fi or wired communication.

[2215] Step 3:

[2216] The server uses a generative AI model to analyze the audio data and suggest optimal volume and frequency spectrum settings.

[2217] Input: Audio data sent to the server

[2218] Output: Optimal volume and frequency spectrum settings

[2219] How it works: A generative AI model on a server analyzes audio data and uses detectors for abnormal sounds and specific frequencies to calculate optimal acoustic settings.

[2220] Step 4:

[2221] The server automatically adjusts the hearing aid settings based on the analysis results.

[2222] Input: Optimal volume and frequency spectrum settings

[2223] Output: Adjusted hearing aid settings

[2224] Specific operation: The server sends adjusted setting information to the user device, and the user device updates the hearing aid settings in real time.

[2225] Step 5:

[2226] Cameras and sensors collect user behavior data in real time.

[2227] Input: User action

[2228] Output: Collected behavioral data

[2229] Specific movements: Cameras and sensors capture the user's movements, such as gait and posture, and store the data in digital form.

[2230] Step 6:

[2231] The collected operation data is sent to a server.

[2232] Input: Collected behavioral data

[2233] Output: Operational data sent to the server

[2234] Specific operation: The user terminal uploads operation data to the server using the communication infrastructure.

[2235] Step 7:

[2236] The server uses a generative AI model to analyze movement data and assess fall risk.

[2237] Input: Operational data sent to the server

[2238] Output: Fall risk assessment results

[2239] Specific movements: A generative AI model analyzes movement data and applies algorithms to detect abnormal movements and assess fall risk.

[2240] Step 8:

[2241] If a high risk of falling is detected, an alert will be issued to the user.

[2242] Input: Fall risk assessment results

[2243] Output: A warning alert to the user

[2244] Specific operation: The device will issue a warning sound or vibration to the user in real time to alert them.

[2245] Step 9:

[2246] A microphone collects the conversational audio and transmits it to a server.

[2247] Input: Conversational audio

[2248] Output: Conversational audio data sent to the server

[2249] What it does: The microphone records the audio of the conversation, stores it digitally, and sends it to a server.

[2250] Step 10:

[2251] The server converts the voice data into text data.

[2252] Input: Conversational audio data sent to the server

[2253] Output: Text data

[2254] What it does: It uses a generative AI model to transcribe audio data and convert it into text format.

[2255] Step 11:

[2256] The server analyzes the text data and extracts important points.

[2257] Input: Text data

[2258] Output: Key points excerpt

[2259] How it works: The generative AI model analyzes the conversation, identifying and extracting keywords and key phrases.

[2260] Step 12:

[2261] The extracted important points are converted into illustrations and displayed on the user's device.

[2262] Input: Key points excerpt

[2263] Output: Visually easy-to-understand illustrations

[2264] Specific operation: The generative AI model creates an illustration based on key points and displays it on the user's device.

[2265] Step 13:

[2266] The camera and microphone collect the user's facial expressions and voice and transmit them to the server.

[2267] Input: User's facial expression and voice data

[2268] Output: Facial expression and voice data sent to the server

[2269] How it works: The camera and microphone record the user's facial expressions and voice in real time and send them digitally to a server.

[2270] Step 14:

[2271] The server uses an emotion recognition algorithm to recognize the user's emotional state.

[2272] Input: Facial expression and voice data sent to the server

[2273] Output: User's emotional state

[2274] How it works: The generative AI model analyzes the user's facial and voice data to identify emotions.

[2275] Step 15:

[2276] The system adjusts its settings based on the user's emotional state and provides the user with information accordingly.

[2277] Input: User's emotional state

[2278] Output: Adjusted system settings and information provided

[2279] How it works: The server calculates the optimal settings based on the user's emotional state and sends that information to the user's device. The device then adjusts the system settings in real time and provides information based on the emotion.

[2280] The above are the specific processing steps.

[2281] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2282] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2283] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2284] [Fourth embodiment]

[2285] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2286] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2287] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2288] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2289] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2290] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2291] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2292] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2293] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2294] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2295] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2296] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2297] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2298] The present invention is an innovative hearing aid system that utilizes generative AI models and is equipped with multiple functions. Specific embodiments are described below.

[2299] Optimizing hearing aid settings

[2300] This hearing aid system has the ability to automatically optimize hearing aid settings based on the user's hearing environment by simply putting on the hearing aid and launching the settings app. When the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal hearing aid settings. The analysis results are sent back to the device, which automatically adjusts the hearing aid settings. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[2301] For example, if a user finds it difficult to hear people during a meeting, they can optimize their hearing aid settings. The hearing aid device then analyzes the sound environment of the meeting, and the AI ​​model proposes optimal volume and filter settings. The hearing aid device then applies the proposed settings, and the user can confirm the actual effect.

[2302] Fall risk detection and prevention

[2303] The system detects and prevents the risk of falls by collecting user movement data in real time and sending it to a server. Cameras and sensors built into the hearing aid device monitor the user's movements and collect movement data. This data is temporarily stored in the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends an alert to the device, which then alerts the user via audio and vibration. Additionally, if the user actually falls, the hearing aid device immediately sends an alert to emergency contacts.

[2304] For example, if a user is about to lose their balance while descending stairs, the hearing aid device will immediately sound an alarm to alert the user, and if they still fall, an alert will be sent to their family or caregiver.

[2305] Transcription and illustration of conversation content

[2306] The system collects conversational audio in real time, transcribes it, and converts it into illustrations to help users understand the content of the conversation. When a user puts on their hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[2307] For example, if a user is having difficulty understanding the conversation while dining with a friend at a restaurant, the hearing aid will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will appear on the screen.

[2308] Summarizing and illustrating online meetings

[2309] The system collects audio from online meetings in real time, converts the content into text, summarizes it, and displays it as illustrations. Users participate in online meetings, and their hearing aids connect to their PCs or smartphones to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen by the hearing aid terminal or a connected device.

[2310] For example, when a user joins a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed, along with an illustration of the new product.

[2311] The above is a detailed description of the mode for carrying out the invention.

[2312] The processing flow will be explained below.

[2313] Optimizing hearing aid settings

[2314] Processing Steps

[2315] Step 1:

[2316] The user puts on the hearing aid and launches the settings app on their smartphone or the hearing aid.

[2317] Step 2:

[2318] The user presses the "Optimize Settings" button in the app.

[2319] Step 3:

[2320] The device (hearing aid or smartphone) records the surrounding sound environment for a few seconds and collects auditory data such as volume and frequency spectrum.

[2321] Step 4:

[2322] The terminal transmits the collected auditory data to a server.

[2323] Step 5:

[2324] Based on the data received by the server, a generative AI model is run, which analyzes the auditory data and suggests optimal settings.

[2325] Step 6:

[2326] The server returns the configuration data including the analysis results to the terminal.

[2327] Step 7:

[2328] The device applies the received configuration data and automatically adjusts the audio settings of the hearing aid.

[2329] Step 8:

[2330] The user will see a "Setup complete" notification and be asked to review their listening progress and provide feedback.

[2331] Step 9:

[2332] If the user instructs fine adjustment as needed, the setting data is sent to the server again and adjustments are made using the same process.

[2333] Fall risk detection and prevention

[2334] Processing Steps

[2335] Step 1:

[2336] The user wears the hearing aid and goes about their daily activities.

[2337] Step 2:

[2338] The device (camera and sensors built into the hearing aid) collects the user's behavior data in real time.

[2339] Step 3:

[2340] The operating data collected by the device is temporarily stored and an initial analysis is performed using an anomaly detection algorithm.

[2341] Step 4:

[2342] The device sends the analysis results to the server and requests a detailed analysis.

[2343] Step 5:

[2344] Based on the data received by the server, a generative AI model is run to assess the risk of falling.

[2345] Step 6:

[2346] The server returns the results of the risk assessment to the terminal.

[2347] Step 7:

[2348] If the device is judged to be at high risk of falling, it will issue an alert to the user via voice or vibration to warn them.

[2349] Step 8:

[2350] If the user falls, the device will detect the fall and automatically send a text message or phone alert to pre-defined emergency contacts.

[2351] Transcription and illustration of conversation content

[2352] Processing Steps

[2353] Step 1:

[2354] The user puts on the hearing aid and begins a conversation.

[2355] Step 2:

[2356] The device (hearing aid) collects conversational audio in real time.

[2357] Step 3:

[2358] The device temporarily stores the voice data and performs initial voice recognition.

[2359] Step 4:

[2360] The terminal transmits the converted text data to the server.

[2361] Step 5:

[2362] Based on the text data received by the server, a generative AI model is run to analyze keywords and important points.

[2363] Step 6:

[2364] Based on the analysis results, the server converts the content into easy-to-understand illustrations.

[2365] Step 7:

[2366] The server sends the generated illustrations and transcription data back to the device.

[2367] Step 8:

[2368] The device (hearing aid or connected smartphone) displays the transcription results and illustrations on the user's screen.

[2369] Summarizing and illustrating online meetings

[2370] Processing Steps

[2371] Step 1:

[2372] A user joins an online meeting.

[2373] Step 2:

[2374] The user connects the conferencing app on their PC or smartphone to their hearing aid.

[2375] Step 3:

[2376] The device (hearing aid or PC / smartphone) collects the conference audio in real time.

[2377] Step 4:

[2378] The device transcribes the audio data and sends the generated text data to the server.

[2379] Step 5:

[2380] Based on the text data received by the server, a generative AI model is run to generate a summary of the meeting contents.

[2381] Step 6:

[2382] Based on the summary results, the server illustrates important points and keywords.

[2383] Step 7:

[2384] The server sends the summary and illustrations back to the device.

[2385] Step 8:

[2386] The device (hearing aid or connected smartphone) displays a summary and illustrations on the user's screen, allowing the user to efficiently understand the meeting content.

[2387] Example 1

[2388] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2389] Conventional hearing aid systems have difficulty in properly adjusting to the user's individual hearing environment and operating conditions. Furthermore, proper settings according to the environment are often performed manually, placing a burden on the user. Furthermore, functions for detecting the risk of falls and assisting in understanding conversation content are lacking, making them insufficient for improving the user's safety and quality of daily life. A comprehensive system that solves these problems is needed.

[2390] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2391] In this invention, the server includes means for analyzing the user's hearing data and proposing optimal hearing aid settings, means for analyzing the user's motion data and assessing the risk of falling, means for analyzing conversational audio, extracting key points and illustrating them, and means for analyzing conference audio, creating summaries and illustrating them, thereby improving the user's hearing environment and safety, and enabling them to understand the content of conversations and conferences.

[2392] "Hearing data" refers to data related to the user's hearing, such as volume and frequency spectrum obtained through a hearing aid.

[2393] "Motion data" is data that includes information about the user's body movements and posture.

[2394] A "server" is a computer system that works in conjunction with hearing aid terminals via a network to analyze and process data.

[2395] A "generative AI model" is a machine learning algorithm that analyzes collected data and generates optimal suggestions and results.

[2396] "Analysis" is the process of investigating and examining collected data in detail using specialized methods to derive useful information and conclusions.

[2397] "Hearing aid settings" are settings that adjust various parameters of the hearing aid (e.g., volume, frequency filters, etc.).

[2398] "Fall risk" is an assessment that indicates the degree of possibility that the user will fall.

[2399] An "alert" is a notification that warns or alerts the user.

[2400] "Conversational voice" refers to the voice that a user utters while communicating with others.

[2401] "Text data" refers to data obtained by converting voice data into text format.

[2402] "Important points" are parts of a conversation or meeting that are judged to be particularly important.

[2403] An "illustration" is an image that visually represents an important point or summary.

[2404] A "summary" is a concise summary of long information such as that from a meeting.

[2405] The present invention is an innovative hearing assistance system that utilizes a generative AI model and is equipped with multiple functions. Specific embodiments are described below.

[2406] Optimizing hearing aid settings

[2407] This hearing assistance system has the function of automatically optimizing settings based on the hearing environment when the user wears the hearing assistance device and launches the settings app on their smartphone. Specifically, when the user presses the "Optimize Settings" button in the app, the device records the surrounding sound environment for a few seconds and collects hearing data such as volume and frequency spectrum. This data is sent to a server, where an AI model generated by the server analyzes the hearing data and suggests optimal settings. The analysis results are sent back to the device, which automatically adjusts the settings of the hearing assistance device. The user is notified that the settings are complete, and can check their hearing condition and make fine adjustments as necessary.

[2408] For example, if a user finds it difficult to hear people during a meeting, they can optimize their settings. The device will then analyze the sound environment of the meeting, and the AI ​​model will suggest optimal volume and filter settings. The device will then apply the suggested settings, and the user can check the actual effect.

[2409] Fall risk detection and prevention

[2410] The system has the ability to detect and prevent the risk of falls by collecting user movement data in real time and sending it to a server. The device's built-in cameras and sensors monitor the user's movements and collect movement data. This data is temporarily stored on the device, where an initial analysis is performed using an anomaly detection algorithm. The analysis results are sent to the server, where a generative AI model performs a more detailed fall risk assessment. If a high risk of falling is detected, the server sends a warning alert to the device, and the device alerts the user via audio and vibration. Additionally, if the user actually falls, the device immediately sends an alert to emergency contacts.

[2411] For example, if a user is about to lose balance while descending stairs, the device will immediately sound an alarm to warn the user, and if the user still falls, an alert will be sent to a family member or caregiver.

[2412] Transcription and illustration of conversation content

[2413] The system collects conversational audio in real time, transcribes it, and converts it into easy-to-understand illustrations. When a user puts on the hearing aid and begins a conversation, the device collects the conversational audio and converts it into text. The converted data is sent to a server, where a generative AI model analyzes it and extracts key points. The extracted content is converted into illustrations, and the device displays the transcription and illustrations on the user's screen.

[2414] For example, if a user is having difficulty understanding the conversation while dining with friends at a restaurant, the hearing aid device will transcribe the conversation and display the analyzed content along with illustrations on the smartphone. For example, if a friend says, "I saw a movie yesterday," illustrations and key words related to the movie will be displayed on the screen.

[2415] Summarizing and illustrating online meetings

[2416] The system has the ability to collect audio from online meetings in real time, convert the content into text, summarize it, and display it as illustrations. Users participate in online meetings, and the hearing aid connects with a PC or smartphone to collect the meeting audio. The audio data is converted into text and sent to a server, where a generative AI model analyzes and summarizes it. The analyzed summary data is then converted into easier-to-understand illustrations, which are then displayed on the user's screen via their terminal or a connected device.

[2417] For example, when a user participates in a business meeting on Zoom, the hearing aid converts the entire meeting content into text, which is then analyzed and summarized by the server and displayed on the smartphone. If the topic of "new product launch plans" comes up during the meeting, a concise summary of the content is displayed on the screen, along with an illustration of the new product.

[2418] Prompt Sentence Examples

[2419] "Please suggest the best assistive hearing device settings for me during the meeting."

[2420] "Send an alert when the user loses balance while descending stairs."

[2421] "Please transcribe the conversation in the restaurant and show it in illustrations."

[2422] "Please summarize and illustrate the contents of the Zoom meeting."

[2423] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2424] Optimizing hearing aid settings

[2425] Step 1:

[2426] The user launches the Settings app on their smartphone and presses the "Optimize Settings" button.

[2427] Input: User actions

[2428] Output: Triggers configuration optimization

[2429] At this point, the setting optimization process is initiated by the user.

[2430] Step 2:

[2431] The device will use the built-in microphone to record the surrounding sound environment for a few seconds.

[2432] Input: Trigger settings optimization

[2433] Output: Sound environment data

[2434] Specifically, the device captures ambient sounds during the meeting.

[2435] Step 3:

[2436] The device extracts auditory data such as volume and frequency spectrum.

[2437] Input: Sound environment data

[2438] Output: Auditory data

[2439] The collected sound environment data is analyzed and the necessary auditory data is extracted.

[2440] Step 4:

[2441] The terminal compresses the auditory data and prepares it for transmission to the server.

[2442] Input: Auditory data

[2443] Output: Compressed data

[2444] Data such as volume and frequency spectrum is compressed and converted into a format that is sent to the server.

[2445] Step 5:

[2446] The terminal transmits the compressed data to the server.

[2447] Input: Compressed data

[2448] Output: Data on the server

[2449] Data is sent from the terminal to the server.

[2450] Step 6:

[2451] The server unpacks the received data and begins analyzing it.

[2452] Input: Compressed data

[2453] Output: Expanded data

[2454] Extract the data and prepare it for analysis.

[2455] Step 7:

[2456] An AI model generated on the server analyzes the deployment data and calculates the optimal settings.

[2457] Input: Expanded data

[2458] Output: Optimal Settings

[2459] An AI model examines the data and determines the optimal hearing aid settings for the user.

[2460] Step 8:

[2461] The server returns the analysis results to the device.

[2462] Input: Optimal Settings

[2463] Output: Configuration data on the terminal

[2464] The server communicates the optimal settings to the device.

[2465] Step 9:

[2466] The settings of the hearing aid device are automatically adjusted based on the analysis results received by the device.

[2467] Input: Setting data

[2468] Output: Calibrated hearing aids

[2469] Your device automatically changes the settings of your hearing aid.

[2470] Step 10:

[2471] The user will be notified that the setup is complete.

[2472] Input: Adjusted hearing aid

[2473] Output: Notification

[2474] The user is notified when the settings are complete, and can make fine adjustments as needed.

[2475] Fall risk detection and prevention

[2476] Step 1:

[2477] Cameras and sensors built into the device monitor and collect information about the user's behavior.

[2478] Input: User action

[2479] Output: Operation data

[2480] Obtain user behavior data in real time.

[2481] Step 2:

[2482] The operating data collected by the device is temporarily saved and an initial analysis is performed.

[2483] Input: Operation data

[2484] Output: Initial analysis results

[2485] An initial analysis is performed using an anomaly detection algorithm on the device.

[2486] Step 3:

[2487] The device sends the initial analysis results to the server.

[2488] Input: Initial analysis results

[2489] Output: Data on the server

[2490] The initial analysis results are sent to the server to prepare for detailed analysis.

[2491] Step 4:

[2492] The server expands the received data and performs a detailed fall risk assessment using a generative AI model.

[2493] Input: Initial analysis results

[2494] Output: Detailed risk assessment

[2495] The server performs detailed analysis and assesses the risk of falling.

[2496] Step 5:

[2497] If a high risk of falling is detected, the server generates an alert to warn the user.

[2498] Input: Detailed risk assessment

[2499] Output: Alert

[2500] The server generates an alert and sends it to the device.

[2501] Step 6:

[2502] The device receives an alert and sends it to the user via sound or vibration.

[2503] Input: Alert

[2504] Output: Caution

[2505] The device will notify the user of the alert by sound or vibration.

[2506] Step 7:

[2507] If the user actually falls, the device will automatically make an emergency call.

[2508] Input: Fall detection

[2509] Output: Emergency alert

[2510] It detects if the user falls and sends an alert to emergency contacts.

[2511] Transcription and illustration of conversation content

[2512] Step 1:

[2513] A user wears a hearing aid and the terminal collects conversational audio.

[2514] Input: Conversational audio

[2515] Output: Audio data

[2516] To acquire a user's conversation voice in real time.

[2517] Step 2:

[2518] The device converts the collected voice data into text.

[2519] Input: Audio data

[2520] Output: Text data

[2521] Voice recognition software is used to convert voice data into text data.

[2522] Step 3:

[2523] Sends text data to the server.

[2524] Input: Text data

[2525] Output: Data on the server

[2526] The text data is sent to the server.

[2527] Step 4:

[2528] An AI model generated on the server analyzes the text data and extracts key points.

[2529] Input: Text data

[2530] Output: Key Points

[2531] An AI model is used to analyze text data and extract key points from the conversation.

[2532] Step 5:

[2533] The server converts the extracted key points into illustrations.

[2534] Input: Key Points

[2535] Output: illustration data

[2536] Convert key points into illustrations to make them easier to understand visually.

[2537] Step 6:

[2538] The device displays the converted illustration and transcription data on the user's screen.

[2539] Input: illustration data, transcription data

[2540] Output: What the user sees on their screen

[2541] The device displays the illustration and transcription results to provide the user.

[2542] Summarizing and illustrating online meetings

[2543] Step 1:

[2544] A user participates in an online conference, and a terminal collects the conference audio.

[2545] Input: Conference audio

[2546] Output: Audio data

[2547] Collect the user's online conference audio.

[2548] Step 2:

[2549] The terminal converts the collected voice data into text data.

[2550] Input: Audio data

[2551] Output: Text data

[2552] Use speech recognition software to convert meeting audio into text data.

[2553] Step 3:

[2554] Sends text data to the server.

[2555] Input: Text data

[2556] Output: Data on the server

[2557] The text data is sent to the server.

[2558] Step 4:

[2559] The AI ​​model generated on the server analyzes the text data and creates a summary.

[2560] Input: Text data

[2561] Output: Summary data

[2562] The AI ​​model analyzes the meeting content and generates a summary.

[2563] Step 5:

[2564] The server converts the summarized content into illustrations.

[2565] Input: Summary data

[2566] Output: illustration data

[2567] Convert summarized meeting content into visually easy-to-understand illustrations.

[2568] Step 6:

[2569] The terminal displays the converted illustration and summary data on the user's screen.

[2570] Input: illustration data, summary data

[2571] Output: What the user sees on their screen

[2572] The terminal displays the illustration and summary content for the user.

[2573] (Application example 1)

[2574] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2575] Conventional hearing aid systems are limited to analyzing voice and motion data and lack optimal functionality for real-world work environments and safety. Noisy factory environments, in particular, can make it difficult to hear voice instructions and increase the risk of workers falling. A method is needed to resolve these issues and improve work efficiency and safety.

[2576] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2577] In this invention, the server

[2578] A means to analyze environmental sounds in the factory in real time and provide voice instructions to workers with optimal volume and sound quality,

[2579] A method for collecting worker movement data in real time using sensors and assessing the risk of falls.

[2580] A means to transcribe voice instructions and meeting content in real time and convert them into easy-to-understand illustrations for display,

[2581] This enables optimization of voice instructions according to the work environment, improvement of worker safety, and visualization of information.

[2582] "Auditory data" is acoustic information obtained through the user's ears, and is data that includes characteristics such as volume and frequency spectrum.

[2583] A "server" is a computer system on a network that collects, analyzes, and stores data.

[2584] An "AI model" is a model that uses machine learning algorithms to analyze data and derive optimal solutions to specific problems.

[2585] "Hearing aid settings" are various adjustment items for assisting the user's hearing, such as the volume and frequency filter settings of the hearing aid.

[2586] "Motion data" is data that includes information about the user's body movements and posture.

[2587] The "fall risk" is an evaluation value of the possibility that the user will fall, and is calculated based on the motion data.

[2588] A "warning" is an alert or notification issued to alert the user.

[2589] "Transcription" is the process of converting audio into text data.

[2590] An "illustration" is a diagram or picture that visually represents information.

[2591] A "sensor" is a device that senses physical data (e.g., motion or sound) and converts that data into an electronic signal.

[2592] "Analysis" is the process of examining collected data in detail to understand its meaning and patterns.

[2593] "Voice instructions" are instructions or commands given to the user using voice.

[2594] The system for realizing the present invention is configured using the following hardware and software.

[2595] Hardware

[2596] 1. Smartphone (e.g. iPhone or Android device)

[2597] 2. Smart glasses (e.g. Microsoft HoloLens)

[2598] 3. Sensors (accelerometer, gyro sensor, camera, etc.)

[2599] software

[2600] 1. Speech recognition library (e.g. Google Speech-to-Text API)

[2601] 2. Machine learning frameworks (e.g., TensorFlow)

[2602] 3. Real-time motion analysis library (e.g., OpenCV)

[2603] System configuration and processing

[2604] 1. Collecting user hearing and movement data:

[2605] The user wears the hearing aid, and the system collects real-time hearing and motion data through motion sensors and microphones, for example, continuously monitoring the worker's movements and surrounding sounds while working in a factory.

[2606] 2. Data transmission and analysis on the server:

[2607] The collected data is sent via smartphones or smart glasses to a server, which receives the collected data and analyzes it using a generative AI model (e.g., GPT-4).

[2608] 3. Recommend and apply optimal settings:

[2609] The server analyzes the data and then suggests optimal volume and filter settings. This makes it easier to hear work instructions, even in noisy factory environments. The analysis results are sent to the hearing aid device, which automatically adjusts the settings.

[2610] 4. Fall risk assessment and alert notification:

[2611] The system evaluates the risk of falls based on the worker's movement data, and if a high risk is detected, a warning is issued via smartphone or smart glasses. If a fall actually occurs, a notification is sent immediately to the manager.

[2612] 5. Transcription and visualization of audio instructions:

[2613] Work instructions and meeting content are transcribed in real time, and a generative AI model extracts key points and converts them into easy-to-understand illustrations. The converted illustrations and transcription data are displayed on the user's smartphone or smart glasses.

[2614] Specific examples

[2615] When workers performing maintenance on equipment in a factory have difficulty hearing instructions due to excessive noise, the system optimizes voice instructions and continuously monitors the risk of falls to enhance safety. Meetings are also transcribed in real time, and a generative AI model summarizes and displays relevant illustrations to enhance worker understanding.

[2616] Prompt Sentence Examples

[2617] Please convert the following audio data into text and summarize it. When doing so, please extract key points so that the summary can be converted into an easy-to-understand illustration.

[2618] Audio data: {Audio data}

[2619] This configuration makes it possible to optimize voice instructions to suit the work environment, improve worker safety, and visualize information, all of which are the goals of the invention.

[2620] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2621] Step 1:

[2622] The user puts on the hearing aid and launches the application installed on the device. The device begins collecting hearing data and movement data. Specifically, it uses the microphone and sensors built into the hearing aid to monitor the user's surrounding sound environment and movement in real time. Input: Environmental sound, movement data. Output: Collected raw data.

[2623] Step 2:

[2624] The device sends the collected auditory and movement data to a server. The data is transmitted over a network using a secure communication protocol (e.g., HTTPS). Input: Collected raw data. Output: Data transmitted to the server.

[2625] Step 3:

[2626] The server processes the received data and analyzes it using a generative AI model. It analyzes the data in real time using a speech recognition library (e.g., Google Speech-to-Text API) and a machine learning framework (e.g., TensorFlow). Input: Transmitted data. Output: Analysis results (optimal hearing aid settings, fall risk assessment, transcription results, etc.).

[2627] Step 4:

[2628] The server then proposes optimal hearing aid settings based on the analysis results. Specifically, the generative AI model calculates volume and filter settings appropriate for the sound environment and sends the settings information to the device. Input: Analysis results. Output: Hearing aid setting information.

[2629] Step 5:

[2630] The device uses the received hearing aid setting information to automatically adjust the hearing aid settings. The processor inside the hearing aid receives this and adjusts the volume and filter settings in real time. Input: Hearing aid setting information. Output: Automatically adjusted hearing aid settings.

[2631] Step 6:

[2632] The device also continues to analyze movement data and issues an alert to the user if it determines there is a high risk of falling. Specifically, it notifies the user by sound warning or vibration. In addition, if a fall actually occurs, it immediately sends a notification to emergency contacts. Input: Movement data analyzed in real time. Output: Warning alert, emergency notification.

[2633] Step 7:

[2634] When a user is participating in a conversation or meeting, the device converts the collected voice data into text and sends the text data to the server. Input: Voice data. Output: Text data.

[2635] Step 8:

[2636] The server receives text data and uses a generative AI model to extract key points and convert them into easy-to-understand...

Claims

1. means for collecting auditory data of a user; means for transmitting the collected auditory data to a server; A means to analyze hearing data using a server-generated AI model and suggest optimal hearing aid settings; a means for automatically adjusting hearing aid settings based on the analysis results; A system including:

2. means for collecting user behavior data in real time; means for transmitting the collected operation data to a server; A means for assessing fall risk using a server-generated AI model; and a means for alerting the user when a high risk of falling is detected; a means of sending an alert to emergency contacts in the event of a fall; The system of claim 1 , comprising:

3. a means for collecting speech in real time; A means for converting the collected voice data into text data; means for transmitting text data to a server; A means of analyzing text data and extracting key points using an AI model generated on the server; A way to convert content containing key points into easy-to-understand illustrations, a means for displaying the converted illustration and the transcription data to a user; The system of claim 1 , comprising:

Images