system

Abstract

We provide the system. [Solution] Methods for capturing ambient sounds with a microphone, A means of sending the acquired audio data to the server, A means of converting audio data into text, A means for analyzing text data and generating a summary, A means of proposing countermeasures based on the summarized text, A system that includes means for notifying the user of summary text and countermeasures.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is very difficult for people with hearing impairments to understand the speech content of nearby people and public broadcasts during communication in the street. Also, this causes inconvenience in life and mobility experiences and a lack of a sense of security. There is a need to solve these problems and provide an environment in which people with hearing impairments can confidently enjoy a safe life and mobility experience.

Means for Solving the Problems

[0005] This invention provides a system that includes means for acquiring ambient sounds with a microphone, means for transmitting the acquired audio data to a server, means for converting the audio data into text, means for analyzing the text data and generating a summary, means for proposing countermeasures based on the summarized text, and means for notifying the user of the summarized text and countermeasures. This system enables hearing-impaired individuals to understand surrounding audio information in real time and act appropriately. In particular, by transmitting audio data to a server using wireless communication and generating summarized text using generative AI, the system achieves efficient and accurate information provision.

[0006] "Environmental sounds" refer to the natural and artificial sounds of the surrounding environment, and are generally a collection of sounds collected through a microphone.

[0007] A "microphone" is a device that converts sound waves into electrical signals and is used to acquire voice data and ambient sounds.

[0008] "Audio data" refers to data that represents sound acquired through a microphone in digital format.

[0009] A "server" is a computer system that provides information and services over a network, and in this invention, it is responsible for processing voice data.

[0010] "Converting speech to text" refers to the process of converting audio data into corresponding strings of characters using speech recognition technology.

[0011] "Text data" refers to information expressed in characters, and in this invention, it is obtained as a result of converting audio data.

[0012] "Analysis" is the process of breaking down data, understanding its content, and extracting its meaning.

[0013] A "summary" is information obtained by shortening the content of text data and extracting the main points.

[0014] "Countermeasures" refer to suggestions for the next actions or solutions that the user should take.

[0015] "Notification" refers to the act of informing a user of information, and in this invention, it is the process of displaying summary text and countermeasures.

[0016] "Wireless communication" refers to a communication method that uses radio waves to send and receive data, and includes Wi-Fi, 4G, and 5G.

[0017] "Generative AI" is a type of artificial intelligence that has algorithms for generating and summarizing text. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention is a system designed to make it easier for people with hearing impairments to understand the speech of people nearby and public broadcasts in urban environments. This system utilizes speech recognition and generative AI to convert ambient sounds into text and suggest appropriate responses to the user.

[0040] Program processing details

[0041] terminal

[0042] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[0043] server

[0044] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[0045] The server then suggests appropriate actions to the user based on the summarized text. This action can also be influenced by the user's settings and past behavior history. Finally, the summarized text and action data are sent to the user's device.

[0046] User

[0047] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Users can then act accordingly.

[0048] Specific example

[0049] Conversation with someone nearby

[0050] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0051] 2. Terminal: Sends the acquired audio data to the server.

[0052] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[0053] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0054] 5. Server: Sends summary text and data for the countermeasure "Next action: Go to a cafe" to the user's terminal.

[0055] 6. Terminal: Display summary text and countermeasures on the user's screen.

[0056] 7. User: Review the displayed information and decide on the next course of action.

[0057] Public announcements at train stations

[0058] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[0059] 2. Terminal: Sends the acquired audio data to the server.

[0060] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[0061] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0062] 5. Server: Sends summary text and data for the countermeasure "Next action: Take the train in 20 minutes" to the user's terminal.

[0063] 6. Terminal: Display summary text and countermeasures on the user's screen.

[0064] 7. User: Review the displayed information and decide on the next course of action.

[0065] Thus, this system enables people with hearing impairments to understand surrounding audio information in real time and take appropriate action.

[0066] The following describes the processing flow.

[0067] Program processing details

[0068] Step 1:

[0069] The device uses a microphone to capture ambient sounds around the user. The microphone is always on, collecting audio data in real time.

[0070] Step 2:

[0071] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[0072] Step 3:

[0073] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[0074] Step 4:

[0075] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[0076] Step 5:

[0077] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[0078] Step 6:

[0079] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[0080] Step 7:

[0081] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[0082] Step 8:

[0083] Based on the summary generated by the server, it proposes appropriate actions for the user. These proposals are customized based on the user's past behavior and configuration information.

[0084] Step 9:

[0085] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[0086] Step 10:

[0087] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[0088] Step 11:

[0089] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[0090] Step 12:

[0091] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[0092] (Example 1)

[0093] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] Recognizing the difficulties hearing-impaired individuals face in communicating in urban areas and gathering information in public places, there is a need to provide a system that allows them to understand surrounding audio information in real time and take appropriate action.

[0095] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0096] In this invention, the server includes means for converting acquired acoustic data into text, means for analyzing the text data and generating a summary, and means for proposing a response based on the generated summary text. This makes it possible for people with hearing impairments to understand information in real time and take appropriate action in various situations they encounter in their daily lives.

[0097] "Ambient sound" refers to acoustic data that includes all sounds and noises generated in the environment.

[0098] A "microphone" is a device that converts acoustic signals into electrical signals.

[0099] "Audio data" refers to digital or analog data of sound acquired by a microphone.

[0100] A "server" is a computer system that receives and processes audio data.

[0101] "Converting to text" refers to the process of analyzing audio data and converting it into text written in natural language.

[0102] "Analyzing and generating a summary" refers to the process of analyzing text data in detail, extracting important information, and summarizing it in a concise form.

[0103] A "response strategy" refers to instructions or guidelines that suggest the next actions a user should take based on analyzed and summarized information.

[0104] "Notifying the user" means displaying the summary and response generated by the server on the user's terminal, thereby informing the user visually or audibly.

[0105] "Wireless communication" is a general term for communication technologies that use radio waves to send and receive data, including Wi-Fi, 4G, and 5G.

[0106] "Generative artificial intelligence" refers to an artificial intelligence system that has the ability to generate and provide information based on given data.

[0107] This invention is a system that acquires ambient sound using a microphone, converts it into text, and generates and provides a summary and countermeasures to the user. Specifically, it is implemented in the following way.

[0108] Terminal role

[0109] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in high-sensitivity microphone that can capture ambient sound in real time. For example, it can capture a conversation like "Let's go to a nearby cafe" when the user is in a cafe. The acquired audio data is transmitted to a server via wireless communication technologies such as Wi-Fi, 4G, or 5G.

[0110] Server Role

[0111] The server is a computer system that receives, analyzes, and processes acoustic data transmitted from terminals. The server converts the acoustic data into text data using a speech recognition engine such as Google® Speech-to-Text API. This text data is then input into a generative AI model (e.g., OpenAI® GPT-4®) to generate summaries and responses.

[0112] As an example of a specific prompt, the format used is: "Summarize the following text and suggest an appropriate action: 'Let's go to a cafe.'" Based on this, the generative AI model generates summaries and responses such as "Suggestion: Go to a cafe" or "Next action: Start moving."

[0113] After the summary text and response are generated, the server sends this data to the terminal. The server efficiently performs these processes using a highly available and scalable cloud infrastructure.

[0114] User roles

[0115] Users can view the summarized text and response sent from the server through their device screen. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the smartphone screen. Based on this information, users decide and take their next action.

[0116] Specific example

[0117] Example 1: Conversation at a cafe

[0118] 1. The device uses its microphone to pick up the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0119] 2. Send the acquired audio data to the server.

[0120] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text data that says, "Want to go get some coffee at a nearby cafe?"

[0121] 4. Use a generative AI model to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0122] 5. The server sends a summary text and a response message to the terminal saying, "Next action: Go to a cafe."

[0123] 6. The terminal displays a summary text and response on the user's screen.

[0124] 7. The user reviews the displayed information and decides on their next course of action.

[0125] Example 2: Public broadcasting at a train station

[0126] 1. The device uses its microphone to pick up the audio of the public announcement at the station, "The train is delayed. The next train will arrive in 20 minutes."

[0127] 2. Send the acquired audio data to the server.

[0128] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text: "The train is delayed. The next train will arrive in 20 minutes."

[0129] 4. Use a generative AI model to summarize the text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0130] 5. The server sends a summary text and a response to the terminal saying, "Next action: Take the train in 20 minutes."

[0131] 6. The terminal displays a summary text and response on the user's screen.

[0132] 7. The user reviews the displayed information and decides on their next course of action.

[0133] In this way, the present invention makes it possible to understand surrounding audio information in real time and take appropriate action.

[0134] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0135] Step 1:

[0136] The device acquires ambient sound using its microphone. Specifically, the device's built-in microphone captures voices such as, "Want to go get some coffee at a nearby cafe?" This audio data becomes the input. The audio data is converted into a digital format and temporarily stored in memory.

[0137] Step 2:

[0138] The terminal transmits the acquired acoustic data to the server using wireless communication. Specifically, it compresses and transmits the acoustic data via Wi-Fi, 4G, or 5G using the HTTP / 2 or WebSocket protocol. The input is the acoustic data, and the output is the data sent to the server.

[0139] Step 3:

[0140] The server receives audio data sent from the terminal. The input is audio data in digital format, which is converted to text using the Google Speech-to-Text API. Specifically, the server analyzes the audio data and generates the text "Want to go for coffee at a nearby cafe?". The output is the generated text data.

[0141] Step 4:

[0142] The server analyzes the generated text data and produces a summary and response. The input is text data, which is input to a generative AI model (e.g., OpenAI GPT-4) using the prompt "Summarize the following text and suggest an appropriate response: 'Shall we go for coffee at a nearby cafe?'" The generative AI model analyzes this text data and generates a summary and response such as "Suggestion to go to a cafe" or "Next action: Go to a cafe." The output is the summarized text and the response.

[0143] Step 5:

[0144] The server sends the generated summary text and response to the terminal. Specifically, it uses HTTP / 2 or WebSocket to send the generated data to the terminal. The input is the summary text and response, and the output is the data sent to the terminal.

[0145] Step 6:

[0146] The terminal receives a summary text and response sent from the server and displays them to the user. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the terminal screen. The input is the data received from the server, and the output is the content displayed on the terminal screen.

[0147] Step 7:

[0148] The user reviews the summary text and response displayed on the device screen and decides on their next action based on that. For example, they might see the suggestion to "go to a cafe" and decide to go to a cafe. The input is the information displayed on the device screen, and the output is the user's action.

[0149] (Application Example 1)

[0150] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0151] Conventional systems have made it difficult for hearing-impaired individuals to acquire real-time audio information from inside and outside vehicles while traveling and to take appropriate action. In particular, when using autonomous vehicles, there was a lack of support for recognizing traffic information and vehicle announcements and acting accordingly. Therefore, there was a need for a means to enable hearing-impaired individuals to travel safely and efficiently.

[0152] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0153] In this invention, the server includes means for acquiring ambient sounds, means for transmitting the acquired audio data to a remote server, means for converting the audio data into text, means for analyzing the text data and generating a summary, means for proposing countermeasures based on the summary text, and means for notifying a display device of the summary text including the countermeasures. This makes it possible for hearing-impaired individuals to acquire audio information from inside and outside the vehicle in real time, even while using an autonomous vehicle, and to take appropriate action.

[0154] "Ambient sounds" refer to all audio signals occurring in the surrounding environment. These audio signals include traffic information, in-vehicle announcements, and people's conversations.

[0155] "Means" refers to specific methods or technical devices used to achieve a particular objective. In this case, it includes devices and systems that perform functions such as voice acquisition, transmission, conversion, analysis, and notification.

[0156] "Acquiring" refers to the process of receiving and recording audio data.

[0157] A "remote server" refers to a remote computing system that can be accessed via the cloud or a network.

[0158] "To send" refers to the operation of transferring data from one point to another.

[0159] "Audio data" refers to data that represents audio in digital format.

[0160] "Converting to text" refers to the process of converting audio data into text-based data.

[0161] "Analyzing" refers to the process of examining data in detail to understand its meaning and intent.

[0162] A "summary" refers to a concise compilation of the main points of the original text.

[0163] "Generating" refers to the process of creating new data or information.

[0164] "Countermeasures" refer to appropriate guidelines or methods of action for a specific situation.

[0165] "Notifying" refers to the process of informing a user of information.

[0166] A "system" refers to a collection of multiple elements that are interconnected and perform specific functions.

[0167] This invention provides a system that acquires audio information inside and outside a vehicle in real time and suggests appropriate countermeasures when a person with a hearing impairment uses an autonomous vehicle. This system acquires ambient sounds with a microphone, transmits the audio data to a remote server, converts it to text, analyzes and summarizes it, and then suggests countermeasures. This system enables the user to efficiently understand surrounding audio information and take appropriate action.

[0168] Hardware and software to be used

[0169] High-performance microphone: Directional microphone

[0170] Network communication: Wireless communication technology (such as 5G modems)

[0171] Speech recognition engine: Software that converts speech into text (e.g., Google Cloud Speech-to-Text API)

[0172] Generative artificial intelligence: Software that performs text analysis and summary generation (e.g., OpenAI GPT-3®)

[0173] Database: A system that stores user settings and activity history (e.g., MongoDB)

[0174] Vehicle-mounted display: A display for showing results (e.g., touchscreen)

[0175] Processing flow

[0176] Voice acquisition and transmission

[0177] The terminal uses a microphone to capture audio from inside and outside the vehicle in real time. The captured audio data is transmitted to a remote server via wireless communication.

[0178] Speech recognition and text conversion

[0179] The server converts the received audio data into text data using a speech recognition engine. This text data includes traffic information, vehicle announcements, and people's conversations.

[0180] Contextual analysis and summary generation

[0181] Generative artificial intelligence (e.g., OpenAI GPT-3) analyzes text converted from audio data and summarizes important information. This allows users to quickly understand the essential content they need while eliminating redundant information.

[0182] Proposed countermeasures and display

[0183] The server suggests appropriate countermeasures to the user based on the summarized text. Generative artificial intelligence is used to suggest these countermeasures, adapting them based on the user's settings and past behavior history. Finally, the summarized text and suggested countermeasures are displayed on the vehicle's screen.

[0184] Specific example

[0185] 1. Obtaining external traffic information

[0186] The device uses its microphone to pick up an announcement saying, "The road is under construction. Please be careful," and sends the data to the server.

[0187] The server receives the audio data and uses a speech recognition engine to convert it into text data that says, "The road is under construction."

[0188] Generative artificial intelligence analyzes this and generates the summary "road construction."

[0189] The server suggests a "detour" as a countermeasure and displays a summary and the countermeasure on the vehicle's display.

[0190] 2. Example of a prompt statement

[0191] "Please summarize the following external audio message and suggest appropriate countermeasures. Audio: 'Road under construction. Please be careful.'"

[0192] This allows people with hearing impairments to obtain real-time traffic information and respond safely.

[0193] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0194] Step 1:

[0195] The device acquires ambient sounds inside and outside the vehicle in real time using a high-performance microphone. The input is ambient sound signals, and the output is digital audio data. Specifically, the microphone converts sound waves into electrical signals, and a digital-to-analog converter inside the device converts these back into digital data.

[0196] Step 2:

[0197] The terminal acquires digital voice data and transmits it to a remote server using wireless communication technology (e.g., a 5G modem). The input is digital voice data, and the output is the data sent to the server. Specifically, the terminal converts the data into packets via the 5G modem and sends them to the server.

[0198] Step 3:

[0199] The server converts the received digital audio data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is audio data, and the output is text data. Specifically, the speech recognition engine analyzes the features of the audio signal and generates the corresponding text.

[0200] Step 4:

[0201] The server uses generative artificial intelligence (e.g., OpenAI GPT-3) to analyze text data and summarize important information. The input is text data, and the output is a summarized text. Specifically, the generative AI understands the meaning of the text, extracts important keywords and context, and generates a summarized text.

[0202] Step 5:

[0203] The server proposes appropriate countermeasures to the user based on the summary text. The input is the summary text and the user's settings and behavior history, and the output is the countermeasures. Specifically, the server refers to information stored in a database (e.g., MongoDB) and formulates appropriate countermeasures using generative artificial intelligence.

[0204] Step 6:

[0205] The server notifies the vehicle's display of the summary text and proposed countermeasures. The input is the summary text and countermeasure data, and the output is what is displayed on the screen. Specifically, the server generates the display data and sends it to the vehicle's infotainment system.

[0206] Step 7:

[0207] The user reviews the summary text and countermeasures notified via the vehicle's display and decides on their next action. The input is the information displayed on the screen, and the output is the user's action selection. Specifically, the user operates the touchscreen on the display to select a countermeasure and then acts accordingly.

[0208] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0209] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions to adjust responses accordingly. This system utilizes speech recognition, generative AI, and an emotion engine to convert ambient sounds into text and then proposes appropriate responses to the user based on that text.

[0210] Program processing details

[0211] terminal

[0212] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[0213] server

[0214] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[0215] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to acquire emotion data. This emotion data has also been learned from past emotion data, allowing it to recognize the user's current emotions more accurately.

[0216] server

[0217] Based on the generated summary text and sentiment data, the server suggests appropriate actions for the user. These suggestions are tailored to the user's emotional state. For example, if the user is stressed, suggestions that promote relaxation can be offered. Finally, the summary text and suggested actions are sent to the user's device.

[0218] User

[0219] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Furthermore, if the user needs to relax, a suggestion such as "How about going to a cafe to relax?" will be made. Users can then act accordingly.

[0220] Specific example

[0221] Conversation with someone nearby

[0222] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0223] 2. Terminal: Sends the acquired audio data to the server.

[0224] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[0225] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0226] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is experiencing stress.

[0227] 6. Server: Based on emotional data, it generates a response such as "Next action: Go to a cafe, but spend the time relaxing."

[0228] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[0229] 8. Terminal: Display summary text and countermeasures on the user's screen.

[0230] 9. User: Review the displayed information and decide on the next course of action.

[0231] Public announcements at train stations

[0232] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[0233] 2. Terminal: Sends the acquired audio data to the server.

[0234] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[0235] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0236] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is anxious.

[0237] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[0238] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[0239] 8. Terminal: Display summary text and countermeasures on the user's screen.

[0240] 9. User: Review the displayed information and decide on the next course of action.

[0241] Thus, this system enables people with hearing impairments to understand surrounding auditory information in real time and to respond appropriately according to their emotions.

[0242] The following describes the processing flow.

[0243] Program processing details

[0244] Step 1:

[0245] The device uses a microphone to capture ambient sounds around the user. The microphone is always on, collecting audio data in real time.

[0246] Step 2:

[0247] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[0248] Step 3:

[0249] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[0250] Step 4:

[0251] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[0252] Step 5:

[0253] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[0254] Step 6:

[0255] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[0256] Step 7:

[0257] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[0258] Step 8:

[0259] The server activates the emotion engine to analyze the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotions from the specified input data.

[0260] Step 9:

[0261] The server analyzes emotional data to determine, for example, levels of stress and relaxation. This process also utilizes the learning results from past emotional data.

[0262] Step 10:

[0263] The server proposes appropriate actions for the user based on the generated summary text and sentiment data. These proposals are adjusted according to the user's current emotional state.

[0264] Step 11:

[0265] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[0266] Step 12:

[0267] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[0268] Step 13:

[0269] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[0270] Step 14:

[0271] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[0272] Specific example

[0273] Conversation with someone nearby

[0274] 1. Step 1: The device uses its microphone to pick up the voice of someone nearby saying, "Want to go get some coffee at a nearby cafe?"

[0275] 2. Step 2: The device saves the acquired audio data to temporary memory and then packets it.

[0276] 3. Step 3: The terminal sends the packetized voice data to the server using wireless communication.

[0277] 4. Step 4: The server reconstructs the received audio data and saves it as an audio file.

[0278] 5. Step 5: The server activates the speech recognition engine to analyze the saved audio file and convert it into text data.

[0279] 6. Step 6: The server inputs the text data into the natural language processing engine for analysis. Analysis includes understanding the context and extracting key points.

[0280] 7. Step 7: The server uses generative AI to generate a summary. It generates the suggestion "Let's go to a cafe."

[0281] 8. Step 8: The server analyzes the user's facial expressions and voice tone using an emotion engine.

[0282] 9. Step 9: The server analyzes the emotional data and determines that the user needs to relax.

[0283] 10. Step 10: Based on the summary text "Proposal to go to a café", the server generates a countermeasure "Next action: Go to a café. Spend time for relaxation" that reflects the emotion data.

[0284] 11. Step 11: The server packetizes the summary text and the countermeasure and sends them to the terminal.

[0285] 12. Step 12: The terminal decrypts the received data and stores it in the local memory.

[0286] 13. Step 13: The terminal displays the summary text and the countermeasure on the screen.

[0287] 14. Step 14: The user checks the information on the screen and determines the next action.

[0288] (Example 2)

[0289] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart device 14 is referred to as the "terminal".

[0290] Conventionally, systems for facilitating communication of hearing-impaired people in the street have had problems such as low recognition accuracy of environmental sounds and text conversion, and no appropriate responses being proposed according to the emotional state. In addition, there have been many systems that are difficult to present countermeasures in real time and cannot sufficiently capture the emotional changes of users. As a result, a situation has occurred in which it is difficult for hearing-impaired people to take appropriate actions.

[0291] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following respective means.

[0292] In this invention, the server includes means for acquiring ambient sounds with an audio input device, means for transmitting the acquired audio data to an information processing device, means for converting the audio data into language data, means for analyzing the language data and generating a summary, means for proposing countermeasures based on the summary text and the user's emotional state, means for notifying the user of the summary text and countermeasures, and means for analyzing the user's emotional state. This enables hearing-impaired individuals to recognize ambient sounds in real time and respond appropriately according to their emotions.

[0293] "Environmental sounds" refer to sounds that occur in everyday life, such as the voices of people around you or public broadcasts.

[0294] A "voice input device" is a device that includes a microphone or other voice input means for acquiring voice data.

[0295] An "information processing device" refers to a computer or server with data processing capabilities, and is a device used to process acquired audio data.

[0296] "Linguistic data" refers to text data and linguistic information obtained by analyzing audio data.

[0297] "Analysis" is the process of understanding the meaning and context based on acquired data, and then further processing the understood information.

[0298] "Summarization" refers to the process of extracting important content and key points from analyzed information in a concise manner.

[0299] "Generation" refers to creating new data or information based on given data.

[0300] "Recommendations" refer to suggesting actions or countermeasures to users based on analysis results and generated data.

[0301] A "notification" is a means of informing the user of the generated summary text and proposed countermeasures.

[0302] "User" refers to a person with hearing impairment or other users who use this system.

[0303] "Emotional state" refers to the current psychological situation or emotion of the user, including states such as stress, satisfaction, and anxiety.

[0304] "Emotion engine" refers to software or hardware for recognizing the emotional state by analyzing the user's facial expressions and voice tones.

[0305] "Artificial intelligence model" refers to artificial intelligence technology that learns from a vast amount of data to make predictions and generate content, specifically including generative AI.

[0306] This invention is a system designed to facilitate communication for people with hearing impairments in the street. This system is realized by combining a voice input device, an information processing device, a voice recognition engine, a generative AI model, and an emotion engine.

[0307] First, a smartphone or a dedicated device as a terminal is used. The terminal has a built-in microphone to acquire ambient environmental sounds in real time. For example, the terminal records a conversation such as "Don't you want to go to a nearby café to have coffee?" with the microphone. The acquired voice data is transmitted to the server via wireless communication such as Wi-Fi, 4G, or 5G.

[0308] The server passes the received voice data to a voice recognition engine (e.g., general voice recognition software) to convert the voice data into text data. For example, voice data such as "Don't you want to go to a nearby café to have coffee?" is converted into text data.

[0309] Next, the text data is analyzed by a generative AI model (e.g., a well-known generative AI model) to summarize important information. In this process, the generative AI performs appropriate context analysis on the input text data and generates a summary such as "A proposal to go to the café".

[0310] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to understand their emotional state. The emotion engine also refers to past emotional data to more accurately recognize the user's current emotions. For example, if the system detects that the user is feeling stressed, "stress" will be recorded as emotional data.

[0311] The server generates appropriate countermeasures based on this sentiment data and the generated summary text. Using a generative AI model, it suggests countermeasures tailored to the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing."

[0312] Finally, the generated summary text and suggested actions are sent back to the device via wireless communication. The user's device receives this and notifies the user by displaying it on the screen. For example, the device's display might show actions such as "Go to a cafe" and "Spend some time relaxing."

[0313] The user reviews the displayed information and decides on their next action based on it. For example, the user decides whether to follow the suggestion to "go to a cafe" and then takes action.

[0314] Specific example

[0315] Example 1: Conversation with someone nearby

[0316] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0317] 2. Terminal: Sends the acquired audio data to the server. Uses wireless communication such as Wi-Fi.

[0318] 3. Server: Converts audio data into text data using a speech recognition engine.

[0319] 4. Server: Analyzes text data using a generative AI model and summarizes it as "a suggestion to go to a cafe."

[0320] 5. Server: The emotion engine analyzes the user's emotional state and recognizes that the user is experiencing stress.

[0321] 6. Server: Based on sentiment data and summary text, it generates a response: "Next action: Go to a cafe, but spend the time relaxing."

[0322] 7. Server: Sends a summary text and countermeasures to the user's terminal.

[0323] 8. Terminal: Display the summary text and countermeasures on the screen.

[0324] 9. User: Review the displayed information and decide on the next course of action.

[0325] Example 2: Public announcements at train stations

[0326] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[0327] 2. Terminal: Sends the acquired audio data to the server.

[0328] 3. Server: The speech data is converted into text, "The train is delayed. The next train will arrive in 20 minutes," using a speech recognition engine.

[0329] 4. Server: A generative AI model analyzes the text data and summarizes it as, "The train is delayed. The next one will arrive in 20 minutes."

[0330] 5. Server: Uses an emotion engine to analyze the user's emotional state and recognizes that the user is feeling anxious.

[0331] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[0332] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[0333] 8. Terminal: Display summary text and countermeasures on the user's screen.

[0334] 9. User: Review the displayed information and decide on the next course of action.

[0335] In this way, this system supports people with hearing impairments in understanding surrounding sounds in real time and taking appropriate responses based on their emotions.

[0336] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0337] Step 1:

[0338] Terminal: The system acquires ambient sounds using an audio input device (microphone). A microphone built into the user's smartphone or dedicated device captures ambient sounds and converts them into digital audio data. The input is analog ambient sound, and the output is digital audio data.

[0339] Step 2:

[0340] Terminal: Transmits acquired voice data to the information processing device (server) via wireless communication (Wi-Fi, 4G, 5G, etc.). The communication module compresses the voice data and sends it to the server's API endpoint using the appropriate communication protocol. Input is digital voice data, and output is data transmission to the server.

[0341] Step 3:

[0342] Server: Receives audio data sent from the terminal and passes it to the speech recognition engine, converting it into language data (text data). The speech recognition engine analyzes the audio data and generates the corresponding text data. For example, it converts the audio data "Shall we go for coffee at a nearby cafe?" into the text data "Shall we go for coffee at a nearby cafe?". The input is digital audio data, and the output is text data.

[0343] Step 4:

[0344] Server: Uses a generative AI model to analyze text data and summarize important information based on context. The AI ​​model understands the intent and context of the text and generates summarized text such as "a suggestion to go to a cafe." Input is text data, output is summarized text.

[0345] Step 5:

[0346] Server: Uses an emotion engine to analyze the user's facial expressions and voice tone transmitted from the terminal to understand their emotional state. Image data and audio data captured by the user's camera are analyzed to derive emotional states such as stress, satisfaction, and anxiety. Input is image data and audio data, and output is emotion data.

[0347] Step 6:

[0348] Server: Generates appropriate countermeasures based on sentiment data and generated summary text. Using a generative AI model, it suggests the most suitable countermeasure for the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing." Input is sentiment data and summary text, output is the countermeasure.

[0349] Step 7:

[0350] Server: Sends the generated summary text and countermeasures back to the user's terminal via wireless communication. The data transmission module compresses the summary text and countermeasures and sends the data to the terminal's API endpoint. Input is the summary text and countermeasures, and output is data transmission to the terminal.

[0351] Step 8:

[0352] Terminal: Displays the received summary text and countermeasures on the user's screen. The user interface visually displays the text data and notifies the user. For example, the display might show "Let's go to a cafe" and "Spend time relaxing." The input is the summary text and countermeasures, and the output is the display on the screen.

[0353] Step 9:

[0354] User: Review the displayed summary text and suggested actions to determine the next course of action. The user takes a specific action based on the information provided, such as deciding to go to a cafe. The input is the information displayed on the screen, and the output is the user's action.

[0355] In this way, the system allows hearing-impaired individuals to understand surrounding sounds in real time and suggest appropriate responses based on their emotions.

[0356] (Application Example 2)

[0357] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0358] This project aims to address the problem of hearing-impaired individuals having difficulty understanding the speech of people nearby or public broadcasts in public spaces, and the lack of means to appropriately manage the user's emotions and provide appropriate support in such situations.

[0359] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data into text, means for analyzing text data and generating a summary, and means for recognizing the user's emotions and adjusting countermeasures based on acquired emotion data. This enables hearing-impaired individuals to understand surrounding audio information in real time and to respond appropriately according to their emotions.

[0360] "Ambient sounds" refer to all kinds of sounds that occur around the user.

[0361] A "microphone" is a device used to convert sound into electrical signals.

[0362] "Audio data" refers to data that represents audio captured by a microphone in digital format.

[0363] A "server" is a computer device that receives voice data via a communication network and performs analysis processing.

[0364] "Text" refers to audio data converted into written form.

[0365] "Generative AI" refers to artificial intelligence systems that analyze given text data and generate summaries or suggestions.

[0366] "Emotional data" refers to data that indicates the user's emotional state, obtained through the user's facial expressions, tone of voice, and other biosensors.

[0367] "Countermeasures" refer to the next actions or appropriate responses suggested to the user based on text data and sentiment data.

[0368] "Means of notification" refers to a system for visually and audibly conveying summary text and countermeasures to the user.

[0369] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions and adjusts the response accordingly. The following describes a specific embodiment of this system.

[0370] To use the system, users wear smart glasses. These smart glasses have a built-in microphone and camera, which can capture ambient sounds and the user's facial expressions in real time.

[0371] Hardware and software to be used

[0372] Smart glasses: Generally, Google Glass® or Vuzix Blade are used.

[0373] Server: Uses a high-performance computing system to perform speech recognition, generative AI, and emotion recognition.

[0374] Speech recognition engine: Uses Google Cloud Speech-to-Text.

[0375] Generative AI: Utilizes OpenAI GPT.

[0376] Emotion engine: Utilizes Microsoft® Azure® Emotion API.

[0377] System Operation Overview

[0378] 1. Audio acquisition

[0379] The microphone in the smart glasses, which is the terminal device, acquires ambient sounds in real time. This audio data is transmitted to a server using wireless communication such as Bluetooth® or Wi-Fi.

[0380] 2. Speech Recognition

[0381] The server converts the received audio data into text data using a speech recognition engine (Google Cloud Speech-to-Text). This conversion allows the spoken content to be treated as text information.

[0382] 3. Text Analysis and Summarization

[0383] The converted text data is passed to a generation AI (OpenAI GPT) for contextual analysis and summarization. This extracts the most important information from the spoken content and generates a short summary.

[0384] 4. Emotion recognition

[0385] The server receives user facial expression data captured using the smart glasses' camera and retrieves emotion data using the emotion engine (Microsoft Azure Emotion API). This allows the server to evaluate the user's current emotional state.

[0386] 5. Proposed countermeasures

[0387] The server uses generative AI to suggest countermeasures based on summarized text and sentiment data. Specific action suggestions are tailored depending on whether the user is experiencing stress or needs relaxation.

[0388] 6. Notification to the user

[0389] The final summary text and suggested countermeasures are displayed on the smart glasses' screen. This allows users to understand surrounding speech and public announcements in real time and take action according to their own emotional state.

[0390] Specific example

[0391] Conversation with someone nearby

[0392] 1. The server receives the utterance "Want to go for coffee at a nearby cafe?" picked up by the microphone on the smart glasses from a nearby person.

[0393] 2. Convert the audio data into text and use a generation AI to summarize it as "a suggestion to go to a cafe."

[0394] 3. If the user's facial expression indicates they are experiencing stress, a suggestion such as "How about going to a cafe to relax?" will be displayed.

[0395] Public announcements at train stations

[0396] 1. The smart glasses' microphone picks up the announcement, "The train is delayed. The next train will arrive in 20 minutes," and sends it to the server.

[0397] 2. Convert the audio data into text and summarize it as, "The train is delayed. The next one will arrive in 20 minutes."

[0398] 3. If the user is anxious, suggest a course of action such as, "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[0399] Example of a prompt

[0400] Summarize the following statement: "Want to go get some coffee at a nearby cafe?" Summary:

[0401] Suggestion when the user is sad: Suggest going to a cafe. Next action:

[0402] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0403] Step 1:

[0404] The device uses a microphone to acquire ambient sounds in real time. These ambient sounds include conversations and public broadcasts. The acquired audio data is converted into a digital format and transmitted to a server via wireless communication (Wi-Fi or Bluetooth).

[0405] Input: Ambient sounds

[0406] Output: Digital audio data

[0407] Step 2:

[0408] The server converts the received audio data into text using a speech recognition engine (Google Cloud Speech-to-Text). The speech recognition engine analyzes the audio data and represents it as textual information.

[0409] Input: Digital audio data

[0410] Output: Converted text data

[0411] Step 3:

[0412] The server uses a generative AI (OpenAI GPT) to analyze text data and generate a summary. The generative AI performs contextual analysis and extracts important information to generate the summarized text.

[0413] Input: Converted text data

[0414] Output: Summarized text data

[0415] Step 4:

[0416] The device's camera is used to acquire user facial expression data. This data is used to recognize the user's emotions. The acquired facial expression data is transmitted to a server via wireless communication.

[0417] Input: User facial expression data

[0418] Output: Digital facial expression data

[0419] Step 5:

[0420] The server uses an emotion engine (Microsoft Azure Emotion API) to analyze the user's facial expressions and obtain emotional data. The emotion engine performs facial analysis and outputs the user's emotional state as data.

[0421] Input: Digital facial expression data

[0422] Output: Sentiment data

[0423] Step 6:

[0424] The server uses generative AI to suggest appropriate responses based on the summarized text and sentiment data. These responses are adjusted according to the user's emotional state. For example, if the user is feeling stressed, suggestions for relaxation will be offered.

[0425] Input: Summary text, sentiment data

[0426] Output: Proposed countermeasures

[0427] Step 7:

[0428] The device receives the summarized text and countermeasures sent from the server and displays them on the smart glasses' screen. This allows the user to see the summarized information and suggestions in real time and decide on their next course of action.

[0429] Input: Summary text, proposed countermeasures

[0430] Output: Summary information and suggestions displayed on the screen.

[0431] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0432] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0433] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0434] [Second Embodiment]

[0435] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0436] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0437] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0438] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0439] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0440] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0441] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0442] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0443] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0444] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0445] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0446] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0447] This invention is a system designed to make it easier for people with hearing impairments to understand the speech of people nearby and public broadcasts in urban environments. This system utilizes speech recognition and generative AI to convert ambient sounds into text and suggest appropriate responses to the user.

[0448] Program processing details

[0449] terminal

[0450] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[0451] server

[0452] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[0453] The server then suggests appropriate actions to the user based on the summarized text. This action can also be influenced by the user's settings and past behavior history. Finally, the summarized text and action data are sent to the user's device.

[0454] User

[0455] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Users can then act accordingly.

[0456] Specific example

[0457] Conversation with someone nearby

[0458] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0459] 2. Terminal: Sends the acquired audio data to the server.

[0460] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[0461] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0462] 5. Server: Sends summary text and data for the countermeasure "Next action: Go to a cafe" to the user's terminal.

[0463] 6. Terminal: Display summary text and countermeasures on the user's screen.

[0464] 7. User: Review the displayed information and decide on the next course of action.

[0465] Public announcements at train stations

[0466] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[0467] 2. Terminal: Sends the acquired audio data to the server.

[0468] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[0469] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0470] 5. Server: Sends summary text and data for the countermeasure "Next action: Take the train in 20 minutes" to the user's terminal.

[0471] 6. Terminal: Display summary text and countermeasures on the user's screen.

[0472] 7. User: Review the displayed information and decide on the next course of action.

[0473] Thus, this system enables people with hearing impairments to understand surrounding audio information in real time and take appropriate action.

[0474] The following describes the processing flow.

[0475] Program processing details

[0476] Step 1:

[0477] The device uses a microphone to capture ambient sounds around the user. The microphone is always ON, collecting audio data in real time.

[0478] Step 2:

[0479] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[0480] Step 3:

[0481] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[0482] Step 4:

[0483] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[0484] Step 5:

[0485] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[0486] Step 6:

[0487] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[0488] Step 7:

[0489] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[0490] Step 8:

[0491] Based on the summary generated by the server, it proposes appropriate actions for the user. These proposals are customized based on the user's past behavior and configuration information.

[0492] Step 9:

[0493] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[0494] Step 10:

[0495] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[0496] Step 11:

[0497] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[0498] Step 12:

[0499] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[0500] (Example 1)

[0501] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0502] Recognizing the difficulties hearing-impaired individuals face in communicating in urban areas and gathering information in public places, there is a need to provide a system that allows them to understand surrounding audio information in real time and take appropriate action.

[0503] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0504] In this invention, the server includes means for converting acquired acoustic data into text, means for analyzing the text data and generating a summary, and means for proposing a response based on the generated summary text. This makes it possible for people with hearing impairments to understand information in real time and take appropriate action in various situations they encounter in their daily lives.

[0505] "Ambient sound" refers to acoustic data that includes all sounds and noises generated in the environment.

[0506] A "microphone" is a device that converts acoustic signals into electrical signals.

[0507] "Audio data" refers to digital or analog data of sound acquired by a microphone.

[0508] A "server" is a computer system that receives and processes audio data.

[0509] "Converting to text" refers to the process of analyzing audio data and converting it into text written in natural language.

[0510] "Analyzing and generating a summary" refers to the process of analyzing text data in detail, extracting important information, and summarizing it in a concise form.

[0511] A "response strategy" refers to instructions or guidelines that suggest the next actions a user should take based on analyzed and summarized information.

[0512] "Notifying the user" means displaying the summary and response generated by the server on the user's terminal, thereby informing the user visually or audibly.

[0513] "Wireless communication" is a general term for communication technologies that use radio waves to send and receive data, including Wi-Fi, 4G, and 5G.

[0514] "Generative artificial intelligence" refers to an artificial intelligence system that has the ability to generate and provide information based on given data.

[0515] This invention is a system that acquires ambient sound using a microphone, converts it into text, and generates and provides a summary and countermeasures to the user. Specifically, it is implemented in the following way.

[0516] Terminal role

[0517] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in high-sensitivity microphone that can capture ambient sound in real time. For example, it can capture a conversation like "Let's go to a nearby cafe" when the user is in a cafe. The acquired audio data is transmitted to a server via wireless communication technologies such as Wi-Fi, 4G, or 5G.

[0518] Server Role

[0519] The server is a computer system that receives, analyzes, and processes acoustic data transmitted from terminals. The server converts the acoustic data into text data using a speech recognition engine such as the Google Speech-to-Text API. This text data is then input into a generative AI model (e.g., OpenAI GPT-4) to generate summaries and responses.

[0520] As a concrete example of a prompt, the format used is: "Summarize the following text and suggest an appropriate action: 'Let's go to a cafe.'" Based on this, the generative AI model generates summaries and responses such as "Suggestion: Go to a cafe" or "Next action: Start moving."

[0521] After the summary text and response are generated, the server sends this data to the terminal. The server efficiently performs these processes using a highly available and scalable cloud infrastructure.

[0522] User roles

[0523] Users can view the summarized text and response sent from the server through their device screen. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the smartphone screen. Based on this information, users decide and take their next action.

[0524] Specific example

[0525] Example 1: Conversation at a cafe

[0526] 1. The device uses its microphone to pick up the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0527] 2. Send the acquired audio data to the server.

[0528] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text data that says, "Want to go get some coffee at a nearby cafe?"

[0529] 4. Use a generative AI model to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0530] 5. The server sends a summary text and a response message to the terminal saying, "Next action: Go to a cafe."

[0531] 6. The terminal displays a summary text and response on the user's screen.

[0532] 7. The user reviews the displayed information and decides on their next course of action.

[0533] Example 2: Public broadcasting at a train station

[0534] 1. The device uses its microphone to pick up the audio of the public announcement at the station, "The train is delayed. The next train will arrive in 20 minutes."

[0535] 2. Send the acquired audio data to the server.

[0536] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text: "The train is delayed. The next train will arrive in 20 minutes."

[0537] 4. Use a generative AI model to summarize the text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0538] 5. The server sends a summary text and a response to the terminal saying, "Next action: Take the train in 20 minutes."

[0539] 6. The terminal displays a summary text and response on the user's screen.

[0540] 7. The user reviews the displayed information and decides on their next course of action.

[0541] In this way, the present invention makes it possible to understand surrounding audio information in real time and take appropriate action.

[0542] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0543] Step 1:

[0544] The device acquires ambient sound using its microphone. Specifically, the device's built-in microphone captures voices such as, "Want to go get some coffee at a nearby cafe?" This audio data becomes the input. The audio data is converted into a digital format and temporarily stored in memory.

[0545] Step 2:

[0546] The terminal transmits the acquired acoustic data to the server using wireless communication. Specifically, it compresses and transmits the acoustic data via Wi-Fi, 4G, or 5G using the HTTP / 2 or WebSocket protocol. The input is the acoustic data, and the output is the data sent to the server.

[0547] Step 3:

[0548] The server receives audio data sent from the terminal. The input is audio data in digital format, which is converted to text using the Google Speech-to-Text API. Specifically, the server analyzes the audio data and generates the text "Want to go for coffee at a nearby cafe?". The output is the generated text data.

[0549] Step 4:

[0550] The server analyzes the generated text data and produces a summary and response. The input is text data, which is input to a generative AI model (e.g., OpenAI GPT-4) using the prompt "Summarize the following text and suggest an appropriate response: 'Shall we go for coffee at a nearby cafe?'" The generative AI model analyzes this text data and generates a summary and response such as "Suggestion to go to a cafe" or "Next action: Go to a cafe." The output is the summarized text and the response.

[0551] Step 5:

[0552] The server sends the generated summary text and response to the terminal. Specifically, it uses HTTP / 2 or WebSocket to send the generated data to the terminal. The input is the summary text and response, and the output is the data sent to the terminal.

[0553] Step 6:

[0554] The terminal receives a summary text and response sent from the server and displays them to the user. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the terminal screen. The input is the data received from the server, and the output is the content displayed on the terminal screen.

[0555] Step 7:

[0556] The user reviews the summary text and response displayed on the device screen and decides on their next action based on that. For example, they might see the suggestion to "go to a cafe" and decide to go to a cafe. The input is the information displayed on the device screen, and the output is the user's action.

[0557] (Application Example 1)

[0558] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0559] Conventional systems have made it difficult for hearing-impaired individuals to acquire real-time audio information from inside and outside vehicles while traveling and to take appropriate action. In particular, when using autonomous vehicles, there was a lack of support for recognizing traffic information and vehicle announcements and acting accordingly. Therefore, there was a need for a means to enable hearing-impaired individuals to travel safely and efficiently.

[0560] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0561] In this invention, the server includes means for acquiring ambient sounds, means for transmitting the acquired audio data to a remote server, means for converting the audio data into text, means for analyzing the text data and generating a summary, means for proposing countermeasures based on the summary text, and means for notifying a display device of the summary text including the countermeasures. This makes it possible for hearing-impaired individuals to acquire audio information from inside and outside the vehicle in real time, even while using an autonomous vehicle, and to take appropriate action.

[0562] "Ambient sounds" refer to all audio signals occurring in the surrounding environment. These audio signals include traffic information, in-vehicle announcements, and people's conversations.

[0563] "Means" refers to specific methods or technical devices used to achieve a particular objective. In this case, it includes devices and systems that perform functions such as voice acquisition, transmission, conversion, analysis, and notification.

[0564] "Acquiring" refers to the process of receiving and recording audio data.

[0565] A "remote server" refers to a remote computing system that can be accessed via the cloud or a network.

[0566] "To send" refers to the operation of transferring data from one point to another.

[0567] "Audio data" refers to data that represents audio in digital format.

[0568] "Converting to text" refers to the process of converting audio data into text-based data.

[0569] "Analyzing" refers to the process of examining data in detail to understand its meaning and intent.

[0570] A "summary" refers to a concise compilation of the main points of the original text.

[0571] "Generating" refers to the process of creating new data or information.

[0572] "Countermeasures" refer to appropriate guidelines or methods of action for a specific situation.

[0573] "Notifying" refers to the process of informing a user of information.

[0574] A "system" refers to a collection of multiple elements that are interconnected and perform specific functions.

[0575] This invention provides a system that acquires audio information inside and outside a vehicle in real time and suggests appropriate countermeasures when a person with a hearing impairment uses an autonomous vehicle. This system acquires ambient sounds with a microphone, transmits the audio data to a remote server, converts it to text, analyzes and summarizes it, and then suggests countermeasures. This system enables the user to efficiently understand surrounding audio information and take appropriate action.

[0576] Hardware and software to be used

[0577] High-performance microphone: Directional microphone

[0578] Network communication: Wireless communication technology (such as 5G modems)

[0579] Speech recognition engine: Software that converts speech into text (e.g., Google Cloud Speech-to-Text API)

[0580] Generative artificial intelligence: Software that performs text analysis and summary generation (e.g., OpenAI GPT-3)

[0581] Database: A system that stores user settings and activity history (e.g., MongoDB)

[0582] Vehicle-mounted display: A display for showing results (e.g., touchscreen)

[0583] Processing flow

[0584] Voice acquisition and transmission

[0585] The terminal uses a microphone to capture audio from inside and outside the vehicle in real time. The captured audio data is transmitted to a remote server via wireless communication.

[0586] Speech recognition and text conversion

[0587] The server converts the received audio data into text data using a speech recognition engine. This text data includes traffic information, vehicle announcements, and people's conversations.

[0588] Contextual analysis and summary generation

[0589] Generative artificial intelligence (e.g., OpenAI GPT-3) analyzes text converted from audio data and summarizes important information. This allows users to quickly understand the essential content they need while eliminating redundant information.

[0590] Proposed countermeasures and display

[0591] The server suggests appropriate countermeasures to the user based on the summarized text. Generative artificial intelligence is used to suggest these countermeasures, adapting them based on the user's settings and past behavior history. Finally, the summarized text and suggested countermeasures are displayed on the vehicle's screen.

[0592] Specific example

[0593] 1. Obtaining external traffic information

[0594] The device uses its microphone to pick up an announcement saying, "The road is under construction. Please be careful," and sends the data to the server.

[0595] The server receives the audio data and uses a speech recognition engine to convert it into text data that says, "The road is under construction."

[0596] Generative artificial intelligence analyzes this and generates the summary "road construction."

[0597] The server suggests a "detour" as a countermeasure and displays a summary and the countermeasure on the vehicle's display.

[0598] 2. Example of a prompt statement

[0599] "Please summarize the following external audio message and suggest appropriate countermeasures. Audio: 'Road under construction. Please be careful.'"

[0600] This allows people with hearing impairments to obtain real-time traffic information and respond safely.

[0601] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0602] Step 1:

[0603] The device acquires ambient sounds inside and outside the vehicle in real time using a high-performance microphone. The input is ambient sound signals, and the output is digital audio data. Specifically, the microphone converts sound waves into electrical signals, and a digital-to-analog converter inside the device converts these back into digital data.

[0604] Step 2:

[0605] The terminal acquires digital voice data and transmits it to a remote server using wireless communication technology (e.g., a 5G modem). The input is digital voice data, and the output is the data sent to the server. Specifically, the terminal converts the data into packets via the 5G modem and sends them to the server.

[0606] Step 3:

[0607] The server converts the received digital audio data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is audio data, and the output is text data. Specifically, the speech recognition engine analyzes the features of the audio signal and generates the corresponding text.

[0608] Step 4:

[0609] The server uses generative artificial intelligence (e.g., OpenAI GPT-3) to analyze text data and summarize important information. The input is text data, and the output is a summarized text. Specifically, the generative AI understands the meaning of the text, extracts important keywords and context, and generates a summarized text.

[0610] Step 5:

[0611] The server proposes appropriate countermeasures to the user based on the summary text. The input is the summary text and the user's settings and behavior history, and the output is the countermeasures. Specifically, the server refers to information stored in a database (e.g., MongoDB) and formulates appropriate countermeasures using generative artificial intelligence.

[0612] Step 6:

[0613] The server notifies the vehicle's display of the summary text and proposed countermeasures. The input is the summary text and countermeasure data, and the output is what is displayed on the screen. Specifically, the server generates the display data and sends it to the vehicle's infotainment system.

[0614] Step 7:

[0615] The user reviews the summary text and countermeasures notified via the vehicle's display and decides on their next action. The input is the information displayed on the screen, and the output is the user's action selection. Specifically, the user operates the touchscreen on the display to select a countermeasure and then acts accordingly.

[0616] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0617] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions to adjust responses accordingly. This system utilizes speech recognition, generative AI, and an emotion engine to convert ambient sounds into text and then proposes appropriate responses to the user based on that text.

[0618] Program processing details

[0619] terminal

[0620] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[0621] server

[0622] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[0623] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to acquire emotion data. This emotion data has also learned from past emotion data, allowing it to recognize the user's current emotions more accurately.

[0624] server

[0625] Based on the generated summary text and sentiment data, the server suggests appropriate actions for the user. These suggestions are tailored to the user's emotional state. For example, if the user is stressed, suggestions that promote relaxation can be offered. Finally, the summary text and suggested actions are sent to the user's device.

[0626] User

[0627] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Furthermore, if the user needs to relax, a suggestion such as "How about going to a cafe to relax?" will be made. Users can then act accordingly.

[0628] Specific example

[0629] Conversation with someone nearby

[0630] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0631] 2. Terminal: Sends the acquired audio data to the server.

[0632] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[0633] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0634] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is experiencing stress.

[0635] 6. Server: Based on emotional data, it generates a response such as "Next action: Go to a cafe, but spend the time relaxing."

[0636] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[0637] 8. Terminal: Display summary text and countermeasures on the user's screen.

[0638] 9. User: Review the displayed information and decide on the next course of action.

[0639] Public announcements at train stations

[0640] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[0641] 2. Terminal: Sends the acquired audio data to the server.

[0642] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[0643] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0644] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is anxious.

[0645] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[0646] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[0647] 8. Terminal: Display summary text and countermeasures on the user's screen.

[0648] 9. User: Review the displayed information and decide on the next course of action.

[0649] Thus, this system enables people with hearing impairments to understand surrounding auditory information in real time and to respond appropriately according to their emotions.

[0650] The following describes the processing flow.

[0651] Program processing details

[0652] Step 1:

[0653] The device uses a microphone to capture ambient sounds around the user. The microphone is always ON, collecting audio data in real time.

[0654] Step 2:

[0655] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[0656] Step 3:

[0657] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[0658] Step 4:

[0659] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[0660] Step 5:

[0661] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[0662] Step 6:

[0663] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[0664] Step 7:

[0665] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[0666] Step 8:

[0667] The server activates the emotion engine to analyze the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotions from the specified input data.

[0668] Step 9:

[0669] The server analyzes emotional data to determine, for example, levels of stress and relaxation. This process also utilizes the learning results from past emotional data.

[0670] Step 10:

[0671] The server proposes appropriate actions for the user based on the generated summary text and sentiment data. These proposals are adjusted according to the user's current emotional state.

[0672] Step 11:

[0673] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[0674] Step 12:

[0675] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[0676] Step 13:

[0677] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[0678] Step 14:

[0679] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[0680] Specific example

[0681] Conversation with someone nearby

[0682] 1. Step 1: The device uses its microphone to pick up the voice of someone nearby saying, "Want to go get some coffee at a nearby cafe?"

[0683] 2. Step 2: The device saves the acquired audio data to temporary memory and then packets it.

[0684] 3. Step 3: The terminal sends the packetized voice data to the server using wireless communication.

[0685] 4. Step 4: The server reconstructs the received audio data and saves it as an audio file.

[0686] 5. Step 5: The server activates the speech recognition engine to analyze the saved audio file and convert it into text data.

[0687] 6. Step 6: The server inputs the text data into the natural language processing engine for analysis. Analysis includes understanding the context and extracting key points.

[0688] 7. Step 7: The server uses generative AI to generate a summary. It generates the suggestion "Let's go to a cafe."

[0689] 8. Step 8: The server analyzes the user's facial expressions and voice tone using an emotion engine.

[0690] 9. Step 9: The server analyzes the emotional data and determines that the user needs to relax.

[0691] 10. Step 10: Based on the summary text "Suggestion to go to a cafe," the server generates a response that reflects sentiment data: "Next action: Go to a cafe. Spend time relaxing."

[0692] 11. Step 11: The server packets the summary text and countermeasures and sends them to the terminal.

[0693] 12. Step 12: The terminal decrypts the received data and saves it to local memory.

[0694] 13. Step 13: The device displays a summary text and a solution on the screen.

[0695] 14. Step 14: The user reviews the information on the screen and decides on their next action.

[0696] (Example 2)

[0697] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0698] Traditional systems designed to facilitate communication for the hearing impaired in urban environments have suffered from problems such as low accuracy in recognizing ambient sounds and converting text, and a lack of appropriate responses based on emotional states. Furthermore, many systems struggled to provide real-time solutions and adequately captured changes in the user's emotions. This resulted in situations where the hearing impaired were unable to take appropriate action.

[0699] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0700] In this invention, the server includes means for acquiring ambient sounds with an audio input device, means for transmitting the acquired audio data to an information processing device, means for converting the audio data into language data, means for analyzing the language data and generating a summary, means for proposing countermeasures based on the summary text and the user's emotional state, means for notifying the user of the summary text and countermeasures, and means for analyzing the user's emotional state. This enables hearing-impaired individuals to recognize ambient sounds in real time and respond appropriately according to their emotions.

[0701] "Environmental sounds" refer to sounds that occur in everyday life, such as the voices of people around you or public broadcasts.

[0702] A "voice input device" is a device that includes a microphone or other voice input means for acquiring voice data.

[0703] An "information processing device" refers to a computer or server with data processing capabilities, and is a device used to process acquired audio data.

[0704] "Linguistic data" refers to text data and linguistic information obtained by analyzing audio data.

[0705] "Analysis" is the process of understanding the meaning and context based on acquired data, and then further processing the understood information.

[0706] "Summarization" refers to the process of extracting important content and key points from analyzed information in a concise manner.

[0707] "Generation" refers to creating new data or information based on given data.

[0708] "Recommendations" refer to suggesting actions or countermeasures to users based on analysis results and generated data.

[0709] A "notification" is a means of informing the user of the generated summary text and proposed countermeasures.

[0710] "Users" refers to hearing-impaired individuals and other users who use this system.

[0711] "Emotional state" refers to the user's current psychological situation and emotions, including states such as stress, satisfaction, and anxiety.

[0712] An "emotion engine" refers to software or hardware that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[0713] An "artificial intelligence model" refers to artificial intelligence technology that learns from vast amounts of data to make predictions and generate data, and specifically includes generative AI.

[0714] This invention is a system designed to facilitate communication for people with hearing impairments in urban areas. The system is realized by combining a voice input device, an information processing device, a speech recognition engine, a generative AI model, and an emotion engine.

[0715] First, a smartphone or dedicated device is used as the terminal. The terminal has a built-in microphone that captures ambient sounds in real time. For example, the terminal records a conversation such as "Shall we go for coffee at a nearby cafe?" using its microphone. This captured audio data is transmitted to a server via wireless communication such as Wi-Fi, 4G, or 5G.

[0716] The server passes the received audio data to a speech recognition engine (for example, general speech recognition software), which converts the audio data into text data. For example, the audio data "Want to go get some coffee at a nearby cafe?" is converted into text data.

[0717] Next, the text data is analyzed by a generative AI model (e.g., a well-known generative AI model) to summarize the important information. In this process, the generative AI performs appropriate contextual analysis on the input text data and generates a summary such as "a suggestion to go to a cafe."

[0718] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to understand their emotional state. The emotion engine also refers to past emotional data to more accurately recognize the user's current emotions. For example, if the system detects that the user is feeling stressed, "stress" will be recorded as emotional data.

[0719] The server generates appropriate countermeasures based on this sentiment data and the generated summary text. Using a generative AI model, it suggests countermeasures tailored to the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing."

[0720] Finally, the generated summary text and suggested actions are sent back to the device via wireless communication. The user's device receives this and notifies the user by displaying it on the screen. For example, the device's display might show actions such as "Go to a cafe" and "Spend some time relaxing."

[0721] The user reviews the displayed information and decides on their next action based on it. For example, the user decides whether to follow the suggestion to "go to a cafe" and then takes action.

[0722] Specific example

[0723] Example 1: Conversation with someone nearby

[0724] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0725] 2. Terminal: Sends the acquired audio data to the server. Uses wireless communication such as Wi-Fi.

[0726] 3. Server: Converts audio data into text data using a speech recognition engine.

[0727] 4. Server: Analyzes text data using a generative AI model and summarizes it as "a suggestion to go to a cafe."

[0728] 5. Server: The emotion engine analyzes the user's emotional state and recognizes that the user is experiencing stress.

[0729] 6. Server: Based on sentiment data and summary text, it generates a response: "Next action: Go to a cafe, but spend the time relaxing."

[0730] 7. Server: Sends a summary text and countermeasures to the user's terminal.

[0731] 8. Terminal: Display the summary text and countermeasures on the screen.

[0732] 9. User: Review the displayed information and decide on the next course of action.

[0733] Example 2: Public announcements at train stations

[0734] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[0735] 2. Terminal: Sends the acquired audio data to the server.

[0736] 3. Server: The speech data is converted into text, "The train is delayed. The next train will arrive in 20 minutes," using a speech recognition engine.

[0737] 4. Server: A generative AI model analyzes the text data and summarizes it as, "The train is delayed. The next one will arrive in 20 minutes."

[0738] 5. Server: Uses an emotion engine to analyze the user's emotional state and recognizes that the user is feeling anxious.

[0739] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[0740] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[0741] 8. Terminal: Display summary text and countermeasures on the user's screen.

[0742] 9. User: Review the displayed information and decide on the next course of action.

[0743] In this way, this system supports people with hearing impairments in understanding surrounding sounds in real time and taking appropriate responses based on their emotions.

[0744] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0745] Step 1:

[0746] Terminal: The system acquires ambient sounds using an audio input device (microphone). A microphone built into the user's smartphone or dedicated device captures ambient sounds and converts them into digital audio data. The input is analog ambient sound, and the output is digital audio data.

[0747] Step 2:

[0748] Terminal: Transmits acquired voice data to the information processing device (server) via wireless communication (Wi-Fi, 4G, 5G, etc.). The communication module compresses the voice data and sends it to the server's API endpoint using the appropriate communication protocol. Input is digital voice data, and output is data transmission to the server.

[0749] Step 3:

[0750] Server: Receives audio data sent from the terminal and passes it to the speech recognition engine, converting it into language data (text data). The speech recognition engine analyzes the audio data and generates the corresponding text data. For example, it converts the audio data "Shall we go for coffee at a nearby cafe?" into the text data "Shall we go for coffee at a nearby cafe?". The input is digital audio data, and the output is text data.

[0751] Step 4:

[0752] Server: Uses a generative AI model to analyze text data and summarize important information based on context. The AI ​​model understands the intent and context of the text and generates summarized text such as "a suggestion to go to a cafe." Input is text data, output is summarized text.

[0753] Step 5:

[0754] Server: Uses an emotion engine to analyze the user's facial expressions and voice tone transmitted from the terminal to understand their emotional state. Image data and audio data captured by the user's camera are analyzed to derive emotional states such as stress, satisfaction, and anxiety. Input is image data and audio data, and output is emotion data.

[0755] Step 6:

[0756] Server: Generates appropriate countermeasures based on sentiment data and generated summary text. Using a generative AI model, it suggests the most suitable countermeasure for the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing." Input is sentiment data and summary text, output is the countermeasure.

[0757] Step 7:

[0758] Server: Sends the generated summary text and countermeasures back to the user's terminal via wireless communication. The data transmission module compresses the summary text and countermeasures and sends the data to the terminal's API endpoint. The input is the summary text and countermeasures, and the output is data transmission to the terminal.

[0759] Step 8:

[0760] Terminal: Displays the received summary text and countermeasures on the user's screen. The user interface visually displays the text data and notifies the user. For example, the display might show "Let's go to a cafe" and "Spend time relaxing." The input is the summary text and countermeasures, and the output is the display on the screen.

[0761] Step 9:

[0762] User: Review the displayed summary text and suggested actions to determine the next course of action. The user takes a specific action based on the information provided, such as deciding to go to a cafe. The input is the information displayed on the screen, and the output is the user's action.

[0763] In this way, the system allows hearing-impaired individuals to understand surrounding sounds in real time and suggest appropriate responses based on their emotions.

[0764] (Application Example 2)

[0765] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0766] This project aims to address the problem of hearing-impaired individuals having difficulty understanding the speech of people nearby or public broadcasts in public spaces, and the lack of means to appropriately manage the user's emotions and provide appropriate support in such situations.

[0767] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data into text, means for analyzing text data and generating a summary, and means for recognizing the user's emotions and adjusting countermeasures based on acquired emotion data. This enables hearing-impaired individuals to understand surrounding audio information in real time and to respond appropriately according to their emotions.

[0768] "Ambient sounds" refer to all kinds of sounds that occur around the user.

[0769] A "microphone" is a device used to convert sound into electrical signals.

[0770] "Audio data" refers to data that represents audio captured by a microphone in digital format.

[0771] A "server" is a computer device that receives voice data via a communication network and performs analysis processing.

[0772] "Text" refers to audio data converted into written form.

[0773] "Generative AI" refers to artificial intelligence systems that analyze given text data and generate summaries or suggestions.

[0774] "Emotional data" refers to data that indicates the user's emotional state, obtained through the user's facial expressions, tone of voice, and other biosensors.

[0775] "Countermeasures" refer to the next actions or appropriate responses suggested to the user based on text data and sentiment data.

[0776] A "means of notification" refers to a system for visually and audibly conveying summary text and countermeasures to the user.

[0777] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions and adjusts the response accordingly. The following describes a specific embodiment of this system.

[0778] To use the system, users wear smart glasses. These smart glasses have a built-in microphone and camera, which can capture ambient sounds and the user's facial expressions in real time.

[0779] Hardware and software to be used

[0780] Smart glasses: Generally, Google Glass or Vuzix Blade are used.

[0781] Server: Uses a high-performance computing system to perform speech recognition, generative AI, and emotion recognition.

[0782] Speech recognition engine: Uses Google Cloud Speech-to-Text.

[0783] Generative AI: Utilizes OpenAI GPT.

[0784] Emotion engine: Utilizes the Microsoft Azure Emotion API.

[0785] System Operation Overview

[0786] 1. Audio acquisition

[0787] The microphone in the smart glasses, which is the device itself, captures ambient sounds in real time. This audio data is transmitted to a server using wireless communication such as Bluetooth or Wi-Fi.

[0788] 2. Speech Recognition

[0789] The server converts the received audio data into text data using a speech recognition engine (Google Cloud Speech-to-Text). This conversion allows the spoken content to be treated as text information.

[0790] 3. Text Analysis and Summarization

[0791] The converted text data is passed to a generation AI (OpenAI GPT) for contextual analysis and summarization. This extracts the most important information from the spoken content and generates a short summary.

[0792] 4. Emotion recognition

[0793] The server receives user facial expression data captured using the smart glasses' camera and retrieves emotion data using the emotion engine (Microsoft Azure Emotion API). This allows the server to evaluate the user's current emotional state.

[0794] 5. Proposed countermeasures

[0795] The server uses generative AI to suggest countermeasures based on summarized text and sentiment data. Specific action suggestions are tailored depending on whether the user is experiencing stress or needs relaxation.

[0796] 6. Notification to the user

[0797] The final summary text and suggested countermeasures are displayed on the smart glasses' screen. This allows users to understand surrounding speech and public announcements in real time and take action according to their own emotional state.

[0798] Specific example

[0799] Conversation with someone nearby

[0800] 1. The server receives the utterance "Want to go for coffee at a nearby cafe?" picked up by the microphone on the smart glasses from a nearby person.

[0801] 2. Convert the audio data into text and use a generation AI to summarize it as "a suggestion to go to a cafe."

[0802] 3. If the user's facial expression indicates they are experiencing stress, a suggestion such as "How about going to a cafe to relax?" will be displayed.

[0803] Public announcements at train stations

[0804] 1. The smart glasses' microphone picks up the announcement, "The train is delayed. The next train will arrive in 20 minutes," and sends it to the server.

[0805] 2. Convert the audio data into text and summarize it as, "The train is delayed. The next one will arrive in 20 minutes."

[0806] 3. If the user is anxious, suggest a course of action such as, "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[0807] Example of a prompt

[0808] Summarize the following statement: "Want to go get some coffee at a nearby cafe?" Summary:

[0809] Suggestion when the user is sad: Suggest going to a cafe. Next action:

[0810] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0811] Step 1:

[0812] The device uses a microphone to acquire ambient sounds in real time. These ambient sounds include conversations and public broadcasts. The acquired audio data is converted into a digital format and transmitted to a server via wireless communication (Wi-Fi or Bluetooth).

[0813] Input: Ambient sounds

[0814] Output: Digital audio data

[0815] Step 2:

[0816] The server converts the received audio data into text using a speech recognition engine (Google Cloud Speech-to-Text). The speech recognition engine analyzes the audio data and represents it as textual information.

[0817] Input: Digital audio data

[0818] Output: Converted text data

[0819] Step 3:

[0820] The server uses a generative AI (OpenAI GPT) to analyze text data and generate a summary. The generative AI performs contextual analysis and extracts important information to generate the summarized text.

[0821] Input: Converted text data

[0822] Output: Summarized text data

[0823] Step 4:

[0824] The device's camera is used to acquire user facial expression data. This data is used to recognize the user's emotions. The acquired facial expression data is transmitted to a server via wireless communication.

[0825] Input: User facial expression data

[0826] Output: Digital facial expression data

[0827] Step 5:

[0828] The server uses an emotion engine (Microsoft Azure Emotion API) to analyze the user's facial expressions and obtain emotional data. The emotion engine performs facial analysis and outputs the user's emotional state as data.

[0829] Input: Digital facial expression data

[0830] Output: Sentiment data

[0831] Step 6:

[0832] The server uses generative AI to suggest appropriate responses based on the summarized text and sentiment data. These responses are adjusted according to the user's emotional state. For example, if the user is feeling stressed, suggestions for relaxation will be offered.

[0833] Input: Summary text, sentiment data

[0834] Output: Proposed countermeasures

[0835] Step 7:

[0836] The device receives the summarized text and countermeasures sent from the server and displays them on the smart glasses' screen. This allows the user to see the summarized information and suggestions in real time and decide on their next course of action.

[0837] Input: Summary text, proposed countermeasures

[0838] Output: Summary information and suggestions displayed on the screen.

[0839] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0840] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0841] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0842] [Third Embodiment]

[0843] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0844] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0845] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0846] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0847] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0848] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0849] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0850] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0851] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0852] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0853] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0854] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0855] This invention is a system designed to make it easier for people with hearing impairments to understand the speech of people nearby and public broadcasts in urban environments. This system utilizes speech recognition and generative AI to convert ambient sounds into text and suggest appropriate responses to the user.

[0856] Program processing details

[0857] terminal

[0858] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[0859] server

[0860] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[0861] The server then suggests appropriate actions to the user based on the summarized text. This action can also be influenced by the user's settings and past behavior history. Finally, the summarized text and action data are sent to the user's device.

[0862] User

[0863] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Users can then act accordingly.

[0864] Specific example

[0865] Conversation with someone nearby

[0866] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0867] 2. Terminal: Sends the acquired audio data to the server.

[0868] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[0869] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0870] 5. Server: Sends summary text and data for the countermeasure "Next action: Go to a cafe" to the user's terminal.

[0871] 6. Terminal: Display summary text and countermeasures on the user's screen.

[0872] 7. User: Review the displayed information and decide on the next course of action.

[0873] Public announcements at train stations

[0874] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[0875] 2. Terminal: Sends the acquired audio data to the server.

[0876] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[0877] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0878] 5. Server: Sends summary text and data for the countermeasure "Next action: Take the train in 20 minutes" to the user's terminal.

[0879] 6. Terminal: Display summary text and countermeasures on the user's screen.

[0880] 7. User: Review the displayed information and decide on the next course of action.

[0881] Thus, this system enables people with hearing impairments to understand surrounding audio information in real time and take appropriate action.

[0882] The following describes the processing flow.

[0883] Program processing details

[0884] Step 1:

[0885] The device uses a microphone to capture ambient sounds around the user. The microphone is always ON, collecting audio data in real time.

[0886] Step 2:

[0887] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[0888] Step 3:

[0889] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[0890] Step 4:

[0891] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[0892] Step 5:

[0893] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[0894] Step 6:

[0895] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[0896] Step 7:

[0897] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[0898] Step 8:

[0899] Based on the summary generated by the server, it proposes appropriate actions for the user. These proposals are customized based on the user's past behavior and configuration information.

[0900] Step 9:

[0901] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[0902] Step 10:

[0903] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[0904] Step 11:

[0905] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[0906] Step 12:

[0907] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[0908] (Example 1)

[0909] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0910] Recognizing the difficulties hearing-impaired individuals face in communicating in urban areas and gathering information in public places, there is a need to provide a system that allows them to understand surrounding audio information in real time and take appropriate action.

[0911] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0912] In this invention, the server includes means for converting acquired acoustic data into text, means for analyzing the text data and generating a summary, and means for proposing a response based on the generated summary text. This makes it possible for people with hearing impairments to understand information in real time and take appropriate action in various situations they encounter in their daily lives.

[0913] "Ambient sound" refers to acoustic data that includes all sounds and noises generated in the environment.

[0914] A "microphone" is a device that converts acoustic signals into electrical signals.

[0915] "Audio data" refers to digital or analog data of sound acquired by a microphone.

[0916] A "server" is a computer system that receives and processes audio data.

[0917] "Converting to text" refers to the process of analyzing audio data and converting it into text written in natural language.

[0918] "Analyzing and generating a summary" refers to the process of analyzing text data in detail, extracting important information, and summarizing it in a concise form.

[0919] A "response strategy" refers to instructions or guidelines that suggest the next actions a user should take based on analyzed and summarized information.

[0920] "Notifying the user" means displaying the summary and response generated by the server on the user's terminal, thereby informing the user visually or audibly.

[0921] "Wireless communication" is a general term for communication technologies that use radio waves to send and receive data, including Wi-Fi, 4G, and 5G.

[0922] "Generative artificial intelligence" refers to an artificial intelligence system that has the ability to generate and provide information based on given data.

[0923] This invention is a system that acquires ambient sound using a microphone, converts it into text, and generates and provides a summary and countermeasures to the user. Specifically, it is implemented in the following way.

[0924] Terminal role

[0925] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in high-sensitivity microphone that can capture ambient sound in real time. For example, it can capture a conversation like "Let's go to a nearby cafe" when the user is in a cafe. The acquired audio data is transmitted to a server via wireless communication technologies such as Wi-Fi, 4G, or 5G.

[0926] Server Role

[0927] The server is a computer system that receives, analyzes, and processes acoustic data transmitted from terminals. The server converts the acoustic data into text data using a speech recognition engine such as the Google Speech-to-Text API. This text data is then input into a generative AI model (e.g., OpenAI GPT-4) to generate summaries and responses.

[0928] As a concrete example of a prompt, the format used is: "Summarize the following text and suggest an appropriate action: 'Let's go to a cafe.'" Based on this, the generative AI model generates summaries and responses such as "Suggestion: Go to a cafe" or "Next action: Start moving."

[0929] After the summary text and response are generated, the server sends this data to the terminal. The server efficiently performs these processes using a highly available and scalable cloud infrastructure.

[0930] User roles

[0931] Users can view the summarized text and response sent from the server through their device screen. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the smartphone screen. Based on this information, users decide and take their next action.

[0932] Specific example

[0933] Example 1: Conversation at a cafe

[0934] 1. The device uses its microphone to pick up the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[0935] 2. Send the acquired audio data to the server.

[0936] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text data that says, "Want to go get some coffee at a nearby cafe?"

[0937] 4. Use a generative AI model to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[0938] 5. The server sends a summary text and a response message to the terminal saying, "Next action: Go to a cafe."

[0939] 6. The terminal displays a summary text and response on the user's screen.

[0940] 7. The user reviews the displayed information and decides on their next course of action.

[0941] Example 2: Public broadcasting at a train station

[0942] 1. The device uses its microphone to pick up the audio of the public announcement at the station, "The train is delayed. The next train will arrive in 20 minutes."

[0943] 2. Send the acquired audio data to the server.

[0944] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text: "The train is delayed. The next train will arrive in 20 minutes."

[0945] 4. Use a generative AI model to summarize the text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[0946] 5. The server sends a summary text and a response to the terminal saying, "Next action: Take the train in 20 minutes."

[0947] 6. The terminal displays a summary text and response on the user's screen.

[0948] 7. The user reviews the displayed information and decides on their next course of action.

[0949] In this way, the present invention makes it possible to understand surrounding audio information in real time and take appropriate action.

[0950] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0951] Step 1:

[0952] The device acquires ambient sound using its microphone. Specifically, the device's built-in microphone captures voices such as, "Want to go get some coffee at a nearby cafe?" This audio data becomes the input. The audio data is converted into a digital format and temporarily stored in memory.

[0953] Step 2:

[0954] The terminal transmits the acquired acoustic data to the server using wireless communication. Specifically, it compresses and transmits the acoustic data via Wi-Fi, 4G, or 5G using the HTTP / 2 or WebSocket protocol. The input is the acoustic data, and the output is the data sent to the server.

[0955] Step 3:

[0956] The server receives audio data sent from the terminal. The input is audio data in digital format, which is converted to text using the Google Speech-to-Text API. Specifically, the server analyzes the audio data and generates the text "Want to go for coffee at a nearby cafe?". The output is the generated text data.

[0957] Step 4:

[0958] The server analyzes the generated text data and produces a summary and response. The input is text data, which is input to a generative AI model (e.g., OpenAI GPT-4) using the prompt "Summarize the following text and suggest an appropriate response: 'Shall we go for coffee at a nearby cafe?'" The generative AI model analyzes this text data and generates a summary and response such as "Suggestion to go to a cafe" or "Next action: Go to a cafe." The output is the summarized text and the response.

[0959] Step 5:

[0960] The server sends the generated summary text and response to the terminal. Specifically, it uses HTTP / 2 or WebSocket to send the generated data to the terminal. The input is the summary text and response, and the output is the data sent to the terminal.

[0961] Step 6:

[0962] The terminal receives a summary text and response sent from the server and displays them to the user. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the terminal screen. The input is the data received from the server, and the output is the content displayed on the terminal screen.

[0963] Step 7:

[0964] The user reviews the summary text and response displayed on the device screen and decides on their next action based on that. For example, they might see the suggestion to "go to a cafe" and decide to go to a cafe. The input is the information displayed on the device screen, and the output is the user's action.

[0965] (Application Example 1)

[0966] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0967] Conventional systems have made it difficult for hearing-impaired individuals to acquire real-time audio information from inside and outside vehicles while traveling and to take appropriate action. In particular, when using autonomous vehicles, there was a lack of support for recognizing traffic information and vehicle announcements and acting accordingly. Therefore, there was a need for a means to enable hearing-impaired individuals to travel safely and efficiently.

[0968] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0969] In this invention, the server includes means for acquiring ambient sounds, means for transmitting the acquired audio data to a remote server, means for converting the audio data into text, means for analyzing the text data and generating a summary, means for proposing countermeasures based on the summary text, and means for notifying a display device of the summary text including the countermeasures. This makes it possible for hearing-impaired individuals to acquire audio information from inside and outside the vehicle in real time, even while using an autonomous vehicle, and to take appropriate action.

[0970] "Ambient sounds" refer to all audio signals occurring in the surrounding environment. These audio signals include traffic information, in-vehicle announcements, and people's conversations.

[0971] "Means" refers to specific methods or technical devices used to achieve a particular objective. In this case, it includes devices and systems that perform functions such as voice acquisition, transmission, conversion, analysis, and notification.

[0972] "Acquiring" refers to the process of receiving and recording audio data.

[0973] A "remote server" refers to a remote computing system that can be accessed via the cloud or a network.

[0974] "To send" refers to the operation of transferring data from one point to another.

[0975] "Audio data" refers to data that represents audio in digital format.

[0976] "Converting to text" refers to the process of converting audio data into text-based data.

[0977] "Analyzing" refers to the process of examining data in detail to understand its meaning and intent.

[0978] A "summary" refers to a concise compilation of the main points of the original text.

[0979] "Generating" refers to the process of creating new data or information.

[0980] "Countermeasures" refer to appropriate guidelines or methods of action for a specific situation.

[0981] "Notifying" refers to the process of informing a user of information.

[0982] A "system" refers to a collection of multiple elements that are interconnected and perform specific functions.

[0983] This invention provides a system that acquires audio information inside and outside a vehicle in real time and suggests appropriate countermeasures when a person with a hearing impairment uses an autonomous vehicle. This system acquires ambient sounds with a microphone, transmits the audio data to a remote server, converts it to text, analyzes and summarizes it, and then suggests countermeasures. This system enables the user to efficiently understand surrounding audio information and take appropriate action.

[0984] Hardware and software to be used

[0985] High-performance microphone: Directional microphone

[0986] Network communication: Wireless communication technology (such as 5G modems)

[0987] Speech recognition engine: Software that converts speech into text (e.g., Google Cloud Speech-to-Text API)

[0988] Generative artificial intelligence: Software that performs text analysis and summary generation (e.g., OpenAI GPT-3)

[0989] Database: A system that stores user settings and activity history (e.g., MongoDB)

[0990] Vehicle-mounted display: A display for showing results (e.g., touchscreen)

[0991] Processing flow

[0992] Voice acquisition and transmission

[0993] The terminal uses a microphone to capture audio from inside and outside the vehicle in real time. The captured audio data is transmitted to a remote server via wireless communication.

[0994] Speech recognition and text conversion

[0995] The server converts the received audio data into text data using a speech recognition engine. This text data includes traffic information, vehicle announcements, and people's conversations.

[0996] Contextual analysis and summary generation

[0997] Generative artificial intelligence (e.g., OpenAI GPT-3) analyzes text converted from audio data and summarizes important information. This allows users to quickly understand the essential content they need while eliminating redundant information.

[0998] Proposed countermeasures and display

[0999] The server suggests appropriate countermeasures to the user based on the summarized text. Generative artificial intelligence is used to suggest these countermeasures, adapting them based on the user's settings and past behavior history. Finally, the summarized text and suggested countermeasures are displayed on the vehicle's screen.

[1000] Specific example

[1001] 1. Obtaining external traffic information

[1002] The device uses its microphone to pick up an announcement saying, "The road is under construction. Please be careful," and sends the data to the server.

[1003] The server receives the audio data and uses a speech recognition engine to convert it into text data that says, "The road is under construction."

[1004] Generative artificial intelligence analyzes this and generates the summary "road construction."

[1005] The server suggests a "detour" as a countermeasure and displays a summary and the countermeasure on the vehicle's display.

[1006] 2. Example of a prompt statement

[1007] "Please summarize the following external audio message and suggest appropriate countermeasures. Audio: 'Road under construction. Please be careful.'"

[1008] This allows people with hearing impairments to obtain real-time traffic information and respond safely.

[1009] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1010] Step 1:

[1011] The device acquires ambient sounds inside and outside the vehicle in real time using a high-performance microphone. The input is ambient sound signals, and the output is digital audio data. Specifically, the microphone converts sound waves into electrical signals, and a digital-to-analog converter inside the device converts these back into digital data.

[1012] Step 2:

[1013] The terminal acquires digital voice data and transmits it to a remote server using wireless communication technology (e.g., a 5G modem). The input is digital voice data, and the output is the data sent to the server. Specifically, the terminal converts the data into packets via the 5G modem and sends them to the server.

[1014] Step 3:

[1015] The server converts the received digital audio data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is audio data, and the output is text data. Specifically, the speech recognition engine analyzes the features of the audio signal and generates the corresponding text.

[1016] Step 4:

[1017] The server uses generative artificial intelligence (e.g., OpenAI GPT-3) to analyze text data and summarize important information. The input is text data, and the output is a summarized text. Specifically, the generative AI understands the meaning of the text, extracts important keywords and context, and generates a summarized text.

[1018] Step 5:

[1019] The server proposes appropriate countermeasures to the user based on the summary text. The input is the summary text and the user's settings and behavior history, and the output is the countermeasures. Specifically, the server refers to information stored in a database (e.g., MongoDB) and formulates appropriate countermeasures using generative artificial intelligence.

[1020] Step 6:

[1021] The server notifies the vehicle's display of the summary text and proposed countermeasures. The input is the summary text and countermeasure data, and the output is what is displayed on the screen. Specifically, the server generates the display data and sends it to the vehicle's infotainment system.

[1022] Step 7:

[1023] The user reviews the summary text and countermeasures notified via the vehicle's display and decides on their next action. The input is the information displayed on the screen, and the output is the user's action selection. Specifically, the user operates the touchscreen on the display to select a countermeasure and then acts accordingly.

[1024] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1025] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions to adjust responses accordingly. This system utilizes speech recognition, generative AI, and an emotion engine to convert ambient sounds into text and then proposes appropriate responses to the user based on that text.

[1026] Program processing details

[1027] terminal

[1028] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[1029] server

[1030] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[1031] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to acquire emotion data. This emotion data has also learned from past emotion data, allowing it to recognize the user's current emotions more accurately.

[1032] server

[1033] Based on the generated summary text and sentiment data, the server suggests appropriate actions for the user. These suggestions are tailored to the user's emotional state. For example, if the user is stressed, suggestions that promote relaxation can be offered. Finally, the summary text and suggested actions are sent to the user's device.

[1034] User

[1035] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Furthermore, if the user needs to relax, a suggestion such as "How about going to a cafe to relax?" will be made. Users can then act accordingly.

[1036] Specific example

[1037] Conversation with someone nearby

[1038] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[1039] 2. Terminal: Sends the acquired audio data to the server.

[1040] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[1041] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[1042] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is experiencing stress.

[1043] 6. Server: Based on emotional data, it generates a response such as "Next action: Go to a cafe, but spend the time relaxing."

[1044] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[1045] 8. Terminal: Display summary text and countermeasures on the user's screen.

[1046] 9. User: Review the displayed information and decide on the next course of action.

[1047] Public announcements at train stations

[1048] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[1049] 2. Terminal: Sends the acquired audio data to the server.

[1050] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[1051] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[1052] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is anxious.

[1053] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[1054] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[1055] 8. Terminal: Display summary text and countermeasures on the user's screen.

[1056] 9. User: Review the displayed information and decide on the next course of action.

[1057] Thus, this system enables people with hearing impairments to understand surrounding auditory information in real time and to respond appropriately according to their emotions.

[1058] The following describes the processing flow.

[1059] Program processing details

[1060] Step 1:

[1061] The device uses a microphone to capture ambient sounds around the user. The microphone is always ON, collecting audio data in real time.

[1062] Step 2:

[1063] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[1064] Step 3:

[1065] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[1066] Step 4:

[1067] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[1068] Step 5:

[1069] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[1070] Step 6:

[1071] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[1072] Step 7:

[1073] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[1074] Step 8:

[1075] The server activates the emotion engine to analyze the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotions from the specified input data.

[1076] Step 9:

[1077] The server analyzes emotional data to determine, for example, levels of stress and relaxation. This process also utilizes the learning results from past emotional data.

[1078] Step 10:

[1079] The server proposes appropriate actions for the user based on the generated summary text and sentiment data. These proposals are adjusted according to the user's current emotional state.

[1080] Step 11:

[1081] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[1082] Step 12:

[1083] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[1084] Step 13:

[1085] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[1086] Step 14:

[1087] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[1088] Specific example

[1089] Conversation with someone nearby

[1090] 1. Step 1: The device uses its microphone to pick up the voice of someone nearby saying, "Want to go get some coffee at a nearby cafe?"

[1091] 2. Step 2: The device saves the acquired audio data to temporary memory and then packets it.

[1092] 3. Step 3: The terminal sends the packetized voice data to the server using wireless communication.

[1093] 4. Step 4: The server reconstructs the received audio data and saves it as an audio file.

[1094] 5. Step 5: The server activates the speech recognition engine to analyze the saved audio file and convert it into text data.

[1095] 6. Step 6: The server inputs the text data into the natural language processing engine for analysis. Analysis includes understanding the context and extracting key points.

[1096] 7. Step 7: The server uses generative AI to generate a summary. It generates the suggestion "Let's go to a cafe."

[1097] 8. Step 8: The server analyzes the user's facial expressions and voice tone using an emotion engine.

[1098] 9. Step 9: The server analyzes the emotional data and determines that the user needs to relax.

[1099] 10. Step 10: Based on the summary text "Suggestion to go to a cafe," the server generates a response that reflects sentiment data: "Next action: Go to a cafe. Spend time relaxing."

[1100] 11. Step 11: The server packets the summary text and countermeasures and sends them to the terminal.

[1101] 12. Step 12: The terminal decrypts the received data and saves it to local memory.

[1102] 13. Step 13: The device displays a summary text and a solution on the screen.

[1103] 14. Step 14: The user reviews the information on the screen and decides on their next action.

[1104] (Example 2)

[1105] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1106] Traditional systems designed to facilitate communication for the hearing impaired in urban environments have suffered from problems such as low accuracy in recognizing ambient sounds and converting text, and a lack of appropriate responses based on emotional states. Furthermore, many systems struggled to provide real-time solutions and adequately captured changes in the user's emotions. This resulted in situations where the hearing impaired were unable to take appropriate action.

[1107] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1108] In this invention, the server includes means for acquiring ambient sounds with an audio input device, means for transmitting the acquired audio data to an information processing device, means for converting the audio data into language data, means for analyzing the language data and generating a summary, means for proposing countermeasures based on the summary text and the user's emotional state, means for notifying the user of the summary text and countermeasures, and means for analyzing the user's emotional state. This enables hearing-impaired individuals to recognize ambient sounds in real time and respond appropriately according to their emotions.

[1109] "Environmental sounds" refer to sounds that occur in everyday life, such as the voices of people around you or public broadcasts.

[1110] A "voice input device" is a device that includes a microphone or other voice input means for acquiring voice data.

[1111] An "information processing device" refers to a computer or server with data processing capabilities, and is a device used to process acquired audio data.

[1112] "Linguistic data" refers to text data and linguistic information obtained by analyzing audio data.

[1113] "Analysis" is the process of understanding the meaning and context based on acquired data, and then further processing the understood information.

[1114] "Summarization" refers to the process of extracting important content and key points from analyzed information in a concise manner.

[1115] "Generation" refers to creating new data or information based on given data.

[1116] "Recommendations" refer to suggesting actions or countermeasures to users based on analysis results and generated data.

[1117] A "notification" is a means of informing the user of the generated summary text and proposed countermeasures.

[1118] "Users" refers to hearing-impaired individuals and other users who use this system.

[1119] "Emotional state" refers to the user's current psychological situation and emotions, including states such as stress, satisfaction, and anxiety.

[1120] An "emotion engine" refers to software or hardware that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1121] An "artificial intelligence model" refers to artificial intelligence technology that learns from vast amounts of data to make predictions and generate data, and specifically includes generative AI.

[1122] This invention is a system designed to facilitate communication for people with hearing impairments in urban areas. The system is realized by combining a voice input device, an information processing device, a speech recognition engine, a generative AI model, and an emotion engine.

[1123] First, a smartphone or dedicated device is used as the terminal. The terminal has a built-in microphone that captures ambient sounds in real time. For example, the terminal records a conversation such as "Shall we go for coffee at a nearby cafe?" using its microphone. This captured audio data is transmitted to a server via wireless communication such as Wi-Fi, 4G, or 5G.

[1124] The server passes the received audio data to a speech recognition engine (for example, general speech recognition software), which converts the audio data into text data. For example, the audio data "Want to go get some coffee at a nearby cafe?" is converted into text data.

[1125] Next, the text data is analyzed by a generative AI model (e.g., a well-known generative AI model) to summarize the important information. In this process, the generative AI performs appropriate contextual analysis on the input text data and generates a summary such as "a suggestion to go to a cafe."

[1126] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to understand their emotional state. The emotion engine also refers to past emotional data to more accurately recognize the user's current emotions. For example, if the system detects that the user is feeling stressed, "stress" will be recorded as emotional data.

[1127] The server generates appropriate countermeasures based on this sentiment data and the generated summary text. Using a generative AI model, it suggests countermeasures tailored to the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing."

[1128] Finally, the generated summary text and suggested actions are sent back to the device via wireless communication. The user's device receives this and notifies the user by displaying it on the screen. For example, the device's display might show actions such as "Go to a cafe" and "Spend some time relaxing."

[1129] The user reviews the displayed information and decides on their next action based on it. For example, the user decides whether to follow the suggestion to "go to a cafe" and then takes action.

[1130] Specific example

[1131] Example 1: Conversation with someone nearby

[1132] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[1133] 2. Terminal: Sends the acquired audio data to the server. Uses wireless communication such as Wi-Fi.

[1134] 3. Server: Converts audio data into text data using a speech recognition engine.

[1135] 4. Server: Analyzes text data using a generative AI model and summarizes it as "a suggestion to go to a cafe."

[1136] 5. Server: The emotion engine analyzes the user's emotional state and recognizes that the user is experiencing stress.

[1137] 6. Server: Based on sentiment data and summary text, it generates a response: "Next action: Go to a cafe, but spend the time relaxing."

[1138] 7. Server: Sends a summary text and countermeasures to the user's terminal.

[1139] 8. Terminal: Display the summary text and countermeasures on the screen.

[1140] 9. User: Review the displayed information and decide on the next course of action.

[1141] Example 2: Public announcements at train stations

[1142] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[1143] 2. Terminal: Sends the acquired audio data to the server.

[1144] 3. Server: The speech data is converted into text, "The train is delayed. The next train will arrive in 20 minutes," using a speech recognition engine.

[1145] 4. Server: A generative AI model analyzes the text data and summarizes it as, "The train is delayed. The next one will arrive in 20 minutes."

[1146] 5. Server: Uses an emotion engine to analyze the user's emotional state and recognizes that the user is feeling anxious.

[1147] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[1148] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[1149] 8. Terminal: Display summary text and countermeasures on the user's screen.

[1150] 9. User: Review the displayed information and decide on the next course of action.

[1151] In this way, this system supports people with hearing impairments in understanding surrounding sounds in real time and taking appropriate responses based on their emotions.

[1152] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1153] Step 1:

[1154] Terminal: The system acquires ambient sounds using an audio input device (microphone). A microphone built into the user's smartphone or dedicated device captures ambient sounds and converts them into digital audio data. The input is analog ambient sound, and the output is digital audio data.

[1155] Step 2:

[1156] Terminal: Transmits acquired voice data to the information processing device (server) via wireless communication (Wi-Fi, 4G, 5G, etc.). The communication module compresses the voice data and sends it to the server's API endpoint using the appropriate communication protocol. Input is digital voice data, and output is data transmission to the server.

[1157] Step 3:

[1158] Server: Receives audio data sent from the terminal and passes it to the speech recognition engine, converting it into language data (text data). The speech recognition engine analyzes the audio data and generates the corresponding text data. For example, it converts the audio data "Shall we go for coffee at a nearby cafe?" into the text data "Shall we go for coffee at a nearby cafe?". The input is digital audio data, and the output is text data.

[1159] Step 4:

[1160] Server: Uses a generative AI model to analyze text data and summarize important information based on context. The AI ​​model understands the intent and context of the text and generates summarized text such as "a suggestion to go to a cafe." Input is text data, output is summarized text.

[1161] Step 5:

[1162] Server: Uses an emotion engine to analyze the user's facial expressions and voice tone transmitted from the terminal to understand their emotional state. Image data and audio data captured by the user's camera are analyzed to derive emotional states such as stress, satisfaction, and anxiety. Input is image data and audio data, and output is emotion data.

[1163] Step 6:

[1164] Server: Generates appropriate countermeasures based on sentiment data and generated summary text. Using a generative AI model, it suggests the most suitable countermeasure for the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing." Input is sentiment data and summary text, output is the countermeasure.

[1165] Step 7:

[1166] Server: Sends the generated summary text and countermeasures back to the user's terminal via wireless communication. The data transmission module compresses the summary text and countermeasures and sends the data to the terminal's API endpoint. The input is the summary text and countermeasures, and the output is data transmission to the terminal.

[1167] Step 8:

[1168] Terminal: Displays the received summary text and countermeasures on the user's screen. The user interface visually displays the text data and notifies the user. For example, the display might show "Let's go to a cafe" and "Spend time relaxing." The input is the summary text and countermeasures, and the output is the display on the screen.

[1169] Step 9:

[1170] User: Review the displayed summary text and suggested actions to determine the next course of action. The user takes a specific action based on the information provided, such as deciding to go to a cafe. The input is the information displayed on the screen, and the output is the user's action.

[1171] In this way, the system allows hearing-impaired individuals to understand surrounding sounds in real time and suggest appropriate responses based on their emotions.

[1172] (Application Example 2)

[1173] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1174] This project aims to address the problem of hearing-impaired individuals having difficulty understanding the speech of people nearby or public broadcasts in public spaces, and the lack of means to appropriately manage the user's emotions and provide appropriate support in such situations.

[1175] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data into text, means for analyzing text data and generating a summary, and means for recognizing the user's emotions and adjusting countermeasures based on acquired emotion data. This enables hearing-impaired individuals to understand surrounding audio information in real time and to respond appropriately according to their emotions.

[1176] "Ambient sounds" refer to all kinds of sounds that occur around the user.

[1177] A "microphone" is a device used to convert sound into electrical signals.

[1178] "Audio data" refers to data that represents audio captured by a microphone in digital format.

[1179] A "server" is a computer device that receives voice data via a communication network and performs analysis processing.

[1180] "Text" refers to audio data converted into written form.

[1181] "Generative AI" refers to artificial intelligence systems that analyze given text data and generate summaries or suggestions.

[1182] "Emotional data" refers to data that indicates the user's emotional state, obtained through the user's facial expressions, tone of voice, and other biosensors.

[1183] "Countermeasures" refer to the next actions or appropriate responses suggested to the user based on text data and sentiment data.

[1184] A "means of notification" refers to a system for visually and audibly conveying summary text and countermeasures to the user.

[1185] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions and adjusts the response accordingly. The following describes a specific embodiment of this system.

[1186] To use the system, users wear smart glasses. These smart glasses have a built-in microphone and camera, which can capture ambient sounds and the user's facial expressions in real time.

[1187] Hardware and software to be used

[1188] Smart glasses: Generally, Google Glass or Vuzix Blade are used.

[1189] Server: Uses a high-performance computing system to perform speech recognition, generative AI, and emotion recognition.

[1190] Speech recognition engine: Uses Google Cloud Speech-to-Text.

[1191] Generative AI: Utilizes OpenAI GPT.

[1192] Emotion engine: Utilizes the Microsoft Azure Emotion API.

[1193] System Operation Overview

[1194] 1. Audio acquisition

[1195] The microphone in the smart glasses, which is the device itself, captures ambient sounds in real time. This audio data is transmitted to a server using wireless communication such as Bluetooth or Wi-Fi.

[1196] 2. Speech Recognition

[1197] The server converts the received audio data into text data using a speech recognition engine (Google Cloud Speech-to-Text). This conversion allows the spoken content to be treated as text information.

[1198] 3. Text Analysis and Summarization

[1199] The converted text data is passed to a generation AI (OpenAI GPT) for contextual analysis and summarization. This extracts the most important information from the spoken content and generates a short summary.

[1200] 4. Emotion recognition

[1201] The server receives user facial expression data captured using the smart glasses' camera and retrieves emotion data using the emotion engine (Microsoft Azure Emotion API). This allows the server to evaluate the user's current emotional state.

[1202] 5. Proposed countermeasures

[1203] The server uses generative AI to suggest countermeasures based on summarized text and sentiment data. Specific action suggestions are tailored depending on whether the user is experiencing stress or needs relaxation.

[1204] 6. Notification to the user

[1205] The final summary text and suggested countermeasures are displayed on the smart glasses' screen. This allows users to understand surrounding speech and public announcements in real time and take action according to their own emotional state.

[1206] Specific example

[1207] Conversation with someone nearby

[1208] 1. The server receives the utterance "Want to go for coffee at a nearby cafe?" picked up by the microphone on the smart glasses from a nearby person.

[1209] 2. Convert the audio data into text and use a generation AI to summarize it as "a suggestion to go to a cafe."

[1210] 3. If the user's facial expression indicates they are experiencing stress, a suggestion such as "How about going to a cafe to relax?" will be displayed.

[1211] Public announcements at train stations

[1212] 1. The smart glasses' microphone picks up the announcement, "The train is delayed. The next train will arrive in 20 minutes," and sends it to the server.

[1213] 2. Convert the audio data into text and summarize it as, "The train is delayed. The next one will arrive in 20 minutes."

[1214] 3. If the user is anxious, suggest a course of action such as, "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[1215] Example of a prompt

[1216] Summarize the following statement: "Want to go get some coffee at a nearby cafe?" Summary:

[1217] Suggestion when the user is sad: Suggest going to a cafe. Next action:

[1218] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1219] Step 1:

[1220] The device uses a microphone to acquire ambient sounds in real time. These ambient sounds include conversations and public broadcasts. The acquired audio data is converted into a digital format and transmitted to a server via wireless communication (Wi-Fi or Bluetooth).

[1221] Input: Ambient sounds

[1222] Output: Digital audio data

[1223] Step 2:

[1224] The server converts the received audio data into text using a speech recognition engine (Google Cloud Speech-to-Text). The speech recognition engine analyzes the audio data and represents it as textual information.

[1225] Input: Digital audio data

[1226] Output: Converted text data

[1227] Step 3:

[1228] The server uses a generative AI (OpenAI GPT) to analyze text data and generate a summary. The generative AI performs contextual analysis and extracts important information to generate the summarized text.

[1229] Input: Converted text data

[1230] Output: Summarized text data

[1231] Step 4:

[1232] The device's camera is used to acquire user facial expression data. This data is used to recognize the user's emotions. The acquired facial expression data is transmitted to a server via wireless communication.

[1233] Input: User facial expression data

[1234] Output: Digital facial expression data

[1235] Step 5:

[1236] The server uses an emotion engine (Microsoft Azure Emotion API) to analyze the user's facial expressions and obtain emotional data. The emotion engine performs facial analysis and outputs the user's emotional state as data.

[1237] Input: Digital facial expression data

[1238] Output: Sentiment data

[1239] Step 6:

[1240] The server uses generative AI to suggest appropriate responses based on the summarized text and sentiment data. These responses are adjusted according to the user's emotional state. For example, if the user is feeling stressed, suggestions for relaxation will be offered.

[1241] Input: Summary text, sentiment data

[1242] Output: Proposed countermeasures

[1243] Step 7:

[1244] The device receives the summarized text and countermeasures sent from the server and displays them on the smart glasses' screen. This allows the user to see the summarized information and suggestions in real time and decide on their next course of action.

[1245] Input: Summary text, proposed countermeasures

[1246] Output: Summary information and suggestions displayed on the screen.

[1247] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1248] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1249] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1250] [Fourth Embodiment]

[1251] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1252] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1253] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1254] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1255] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1256] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1257] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1258] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1259] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1260] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1261] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1262] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1263] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1264] This invention is a system designed to make it easier for people with hearing impairments to understand the speech of people nearby and public broadcasts in urban environments. This system utilizes speech recognition and generative AI to convert ambient sounds into text and suggest appropriate responses to the user.

[1265] Program processing details

[1266] terminal

[1267] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[1268] server

[1269] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[1270] The server then suggests appropriate actions to the user based on the summarized text. This action can also be influenced by the user's settings and past behavior history. Finally, the summarized text and action data are sent to the user's device.

[1271] User

[1272] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Users can then act accordingly.

[1273] Specific example

[1274] Conversation with someone nearby

[1275] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[1276] 2. Terminal: Sends the acquired audio data to the server.

[1277] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[1278] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[1279] 5. Server: Sends summary text and data for the countermeasure "Next action: Go to a cafe" to the user's terminal.

[1280] 6. Terminal: Display summary text and countermeasures on the user's screen.

[1281] 7. User: Review the displayed information and decide on the next course of action.

[1282] Public announcements at train stations

[1283] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[1284] 2. Terminal: Sends the acquired audio data to the server.

[1285] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[1286] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[1287] 5. Server: Sends summary text and data for the countermeasure "Next action: Take the train in 20 minutes" to the user's terminal.

[1288] 6. Terminal: Display summary text and countermeasures on the user's screen.

[1289] 7. User: Review the displayed information and decide on the next course of action.

[1290] Thus, this system enables people with hearing impairments to understand surrounding audio information in real time and take appropriate action.

[1291] The following describes the processing flow.

[1292] Program processing details

[1293] Step 1:

[1294] The device uses a microphone to capture ambient sounds around the user. The microphone is always ON, collecting audio data in real time.

[1295] Step 2:

[1296] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[1297] Step 3:

[1298] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[1299] Step 4:

[1300] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[1301] Step 5:

[1302] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[1303] Step 6:

[1304] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[1305] Step 7:

[1306] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[1307] Step 8:

[1308] Based on the summary generated by the server, it proposes appropriate actions for the user. These proposals are customized based on the user's past behavior and configuration information.

[1309] Step 9:

[1310] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[1311] Step 10:

[1312] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[1313] Step 11:

[1314] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[1315] Step 12:

[1316] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[1317] (Example 1)

[1318] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1319] Recognizing the difficulties hearing-impaired individuals face in communicating in urban areas and gathering information in public places, there is a need to provide a system that allows them to understand surrounding audio information in real time and take appropriate action.

[1320] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1321] In this invention, the server includes means for converting acquired acoustic data into text, means for analyzing the text data and generating a summary, and means for proposing a response based on the generated summary text. This makes it possible for people with hearing impairments to understand information in real time and take appropriate action in various situations they encounter in their daily lives.

[1322] "Ambient sound" refers to acoustic data that includes all sounds and noises generated in the environment.

[1323] A "microphone" is a device that converts acoustic signals into electrical signals.

[1324] "Audio data" refers to digital or analog data of sound acquired by a microphone.

[1325] A "server" is a computer system that receives and processes audio data.

[1326] "Converting to text" refers to the process of analyzing audio data and converting it into text written in natural language.

[1327] "Analyzing and generating a summary" refers to the process of analyzing text data in detail, extracting important information, and summarizing it in a concise form.

[1328] A "response strategy" refers to instructions or guidelines that suggest the next actions a user should take based on analyzed and summarized information.

[1329] "Notifying the user" means displaying the summary and response generated by the server on the user's terminal, thereby informing the user visually or audibly.

[1330] "Wireless communication" is a general term for communication technologies that use radio waves to send and receive data, including Wi-Fi, 4G, and 5G.

[1331] "Generative artificial intelligence" refers to an artificial intelligence system that has the ability to generate and provide information based on given data.

[1332] This invention is a system that acquires ambient sound using a microphone, converts it into text, and generates and provides a summary and countermeasures to the user. Specifically, it is implemented in the following way.

[1333] Terminal role

[1334] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in high-sensitivity microphone that can capture ambient sound in real time. For example, it can capture a conversation like "Let's go to a nearby cafe" when the user is in a cafe. The acquired audio data is transmitted to a server via wireless communication technologies such as Wi-Fi, 4G, or 5G.

[1335] Server Role

[1336] The server is a computer system that receives, analyzes, and processes acoustic data transmitted from terminals. The server converts the acoustic data into text data using a speech recognition engine such as the Google Speech-to-Text API. This text data is then input into a generative AI model (e.g., OpenAI GPT-4) to generate summaries and responses.

[1337] As a concrete example of a prompt, the format used is: "Summarize the following text and suggest an appropriate action: 'Let's go to a cafe.'" Based on this, the generative AI model generates summaries and responses such as "Suggestion: Go to a cafe" or "Next action: Start moving."

[1338] After the summary text and response are generated, the server sends this data to the terminal. The server efficiently performs these processes using a highly available and scalable cloud infrastructure.

[1339] User roles

[1340] Users can view the summarized text and response sent from the server through their device screen. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the smartphone screen. Based on this information, users decide and take their next action.

[1341] Specific example

[1342] Example 1: Conversation at a cafe

[1343] 1. The device uses its microphone to pick up the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[1344] 2. Send the acquired audio data to the server.

[1345] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text data that says, "Want to go get some coffee at a nearby cafe?"

[1346] 4. Use a generative AI model to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[1347] 5. The server sends a summary text and a response message to the terminal saying, "Next action: Go to a cafe."

[1348] 6. The terminal displays a summary text and response on the user's screen.

[1349] 7. The user reviews the displayed information and decides on their next course of action.

[1350] Example 2: Public broadcasting at a train station

[1351] 1. The device uses its microphone to pick up the audio of the public announcement at the station, "The train is delayed. The next train will arrive in 20 minutes."

[1352] 2. Send the acquired audio data to the server.

[1353] 3. The server receives the audio data and uses the Google Speech-to-Text API to convert it into text: "The train is delayed. The next train will arrive in 20 minutes."

[1354] 4. Use a generative AI model to summarize the text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[1355] 5. The server sends a summary text and a response to the terminal saying, "Next action: Take the train in 20 minutes."

[1356] 6. The terminal displays a summary text and response on the user's screen.

[1357] 7. The user reviews the displayed information and decides on their next course of action.

[1358] In this way, the present invention makes it possible to understand surrounding audio information in real time and take appropriate action.

[1359] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1360] Step 1:

[1361] The device acquires ambient sound using its microphone. Specifically, the device's built-in microphone captures voices such as, "Want to go get some coffee at a nearby cafe?" This audio data becomes the input. The audio data is converted into a digital format and temporarily stored in memory.

[1362] Step 2:

[1363] The terminal transmits the acquired acoustic data to the server using wireless communication. Specifically, it compresses and transmits the acoustic data via Wi-Fi, 4G, or 5G using the HTTP / 2 or WebSocket protocol. The input is the acoustic data, and the output is the data sent to the server.

[1364] Step 3:

[1365] The server receives audio data sent from the terminal. The input is audio data in digital format, which is converted to text using the Google Speech-to-Text API. Specifically, the server analyzes the audio data and generates the text "Want to go for coffee at a nearby cafe?". The output is the generated text data.

[1366] Step 4:

[1367] The server analyzes the generated text data and produces a summary and response. The input is text data, which is input to a generative AI model (e.g., OpenAI GPT-4) using the prompt "Summarize the following text and suggest an appropriate response: 'Shall we go for coffee at a nearby cafe?'" The generative AI model analyzes this text data and generates a summary and response such as "Suggestion to go to a cafe" or "Next action: Go to a cafe." The output is the summarized text and the response.

[1368] Step 5:

[1369] The server sends the generated summary text and response to the terminal. Specifically, it uses HTTP / 2 or WebSocket to send the generated data to the terminal. The input is the summary text and response, and the output is the data sent to the terminal.

[1370] Step 6:

[1371] The terminal receives a summary text and response sent from the server and displays them to the user. Specifically, messages such as "Suggestion to go to a cafe" and "Next action: Go to a cafe" are displayed on the terminal screen. The input is the data received from the server, and the output is the content displayed on the terminal screen.

[1372] Step 7:

[1373] The user reviews the summary text and response displayed on the device screen and decides on their next action based on that. For example, they might see the suggestion to "go to a cafe" and decide to go to a cafe. The input is the information displayed on the device screen, and the output is the user's action.

[1374] (Application Example 1)

[1375] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1376] Conventional systems have made it difficult for hearing-impaired individuals to acquire real-time audio information from inside and outside vehicles while traveling and to take appropriate action. In particular, when using autonomous vehicles, there was a lack of support for recognizing traffic information and vehicle announcements and acting accordingly. Therefore, there was a need for a means to enable hearing-impaired individuals to travel safely and efficiently.

[1377] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1378] In this invention, the server includes means for acquiring ambient sounds, means for transmitting the acquired audio data to a remote server, means for converting the audio data into text, means for analyzing the text data and generating a summary, means for proposing countermeasures based on the summary text, and means for notifying a display device of the summary text including the countermeasures. This makes it possible for hearing-impaired individuals to acquire audio information from inside and outside the vehicle in real time, even while using an autonomous vehicle, and to take appropriate action.

[1379] "Ambient sounds" refer to all audio signals occurring in the surrounding environment. These audio signals include traffic information, in-vehicle announcements, and people's conversations.

[1380] "Means" refers to specific methods or technical devices used to achieve a particular objective. In this case, it includes devices and systems that perform functions such as voice acquisition, transmission, conversion, analysis, and notification.

[1381] "Acquiring" refers to the process of receiving and recording audio data.

[1382] A "remote server" refers to a remote computing system that can be accessed via the cloud or a network.

[1383] "To send" refers to the operation of transferring data from one point to another.

[1384] "Audio data" refers to data that represents audio in digital format.

[1385] "Converting to text" refers to the process of converting audio data into text-based data.

[1386] "Analyzing" refers to the process of examining data in detail to understand its meaning and intent.

[1387] A "summary" refers to a concise compilation of the main points of the original text.

[1388] "Generating" refers to the process of creating new data or information.

[1389] "Countermeasures" refer to appropriate guidelines or methods of action for a specific situation.

[1390] "Notifying" refers to the process of informing a user of information.

[1391] A "system" refers to a collection of multiple elements that are interconnected and perform specific functions.

[1392] This invention provides a system that acquires audio information inside and outside a vehicle in real time and suggests appropriate countermeasures when a person with a hearing impairment uses an autonomous vehicle. This system acquires ambient sounds with a microphone, transmits the audio data to a remote server, converts it to text, analyzes and summarizes it, and then suggests countermeasures. This system enables the user to efficiently understand surrounding audio information and take appropriate action.

[1393] Hardware and software to be used

[1394] High-performance microphone: Directional microphone

[1395] Network communication: Wireless communication technology (such as 5G modems)

[1396] Speech recognition engine: Software that converts speech into text (e.g., Google Cloud Speech-to-Text API)

[1397] Generative artificial intelligence: Software that performs text analysis and summary generation (e.g., OpenAI GPT-3)

[1398] Database: A system that stores user settings and activity history (e.g., MongoDB)

[1399] Vehicle-mounted display: A display for showing results (e.g., touchscreen)

[1400] Processing flow

[1401] Voice acquisition and transmission

[1402] The terminal uses a microphone to capture audio from inside and outside the vehicle in real time. The captured audio data is transmitted to a remote server via wireless communication.

[1403] Speech recognition and text conversion

[1404] The server converts the received audio data into text data using a speech recognition engine. This text data includes traffic information, vehicle announcements, and people's conversations.

[1405] Contextual analysis and summary generation

[1406] Generative artificial intelligence (e.g., OpenAI GPT-3) analyzes text converted from audio data and summarizes important information. This allows users to quickly understand the essential content they need while eliminating redundant information.

[1407] Proposed countermeasures and display

[1408] The server suggests appropriate countermeasures to the user based on the summarized text. Generative artificial intelligence is used to suggest these countermeasures, adapting them based on the user's settings and past behavior history. Finally, the summarized text and suggested countermeasures are displayed on the vehicle's screen.

[1409] Specific example

[1410] 1. Obtaining external traffic information

[1411] The device uses its microphone to pick up an announcement saying, "The road is under construction. Please be careful," and sends the data to the server.

[1412] The server receives the audio data and uses a speech recognition engine to convert it into text data that says, "The road is under construction."

[1413] Generative artificial intelligence analyzes this and generates the summary "road construction."

[1414] The server suggests a "detour" as a countermeasure and displays a summary and the countermeasure on the vehicle's display.

[1415] 2. Example of a prompt statement

[1416] "Please summarize the following external audio message and suggest appropriate countermeasures. Audio: 'Road under construction. Please be careful.'"

[1417] This allows people with hearing impairments to obtain real-time traffic information and respond safely.

[1418] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1419] Step 1:

[1420] The device acquires ambient sounds inside and outside the vehicle in real time using a high-performance microphone. The input is ambient sound signals, and the output is digital audio data. Specifically, the microphone converts sound waves into electrical signals, and a digital-to-analog converter inside the device converts these back into digital data.

[1421] Step 2:

[1422] The terminal acquires digital voice data and transmits it to a remote server using wireless communication technology (e.g., a 5G modem). The input is digital voice data, and the output is the data sent to the server. Specifically, the terminal converts the data into packets via the 5G modem and sends them to the server.

[1423] Step 3:

[1424] The server converts the received digital audio data into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The input is audio data, and the output is text data. Specifically, the speech recognition engine analyzes the features of the audio signal and generates the corresponding text.

[1425] Step 4:

[1426] The server uses generative artificial intelligence (e.g., OpenAI GPT-3) to analyze text data and summarize important information. The input is text data, and the output is a summarized text. Specifically, the generative AI understands the meaning of the text, extracts important keywords and context, and generates a summarized text.

[1427] Step 5:

[1428] The server proposes appropriate countermeasures to the user based on the summary text. The input is the summary text and the user's settings and behavior history, and the output is the countermeasures. Specifically, the server refers to information stored in a database (e.g., MongoDB) and formulates appropriate countermeasures using generative artificial intelligence.

[1429] Step 6:

[1430] The server notifies the vehicle's display of the summary text and proposed countermeasures. The input is the summary text and countermeasure data, and the output is what is displayed on the screen. Specifically, the server generates the display data and sends it to the vehicle's infotainment system.

[1431] Step 7:

[1432] The user reviews the summary text and countermeasures notified via the vehicle's display and decides on their next action. The input is the information displayed on the screen, and the output is the user's action selection. Specifically, the user operates the touchscreen on the display to select a countermeasure and then acts accordingly.

[1433] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1434] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions to adjust responses accordingly. This system utilizes speech recognition, generative AI, and an emotion engine to convert ambient sounds into text and then proposes appropriate responses to the user based on that text.

[1435] Program processing details

[1436] terminal

[1437] The terminal is a smartphone or dedicated device carried by the user. The terminal has a built-in microphone that can capture ambient sounds in real time. The acquired audio data is transmitted to a server using wireless communication (Wi-Fi, 4G, 5G, etc.).

[1438] server

[1439] The server receives audio data transmitted from the terminal, analyzes it using a speech recognition engine, and converts it into text data. The converted text data is then subjected to contextual analysis by a generative AI to generate text that summarizes the important information.

[1440] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to acquire emotion data. This emotion data has also learned from past emotion data, allowing it to recognize the user's current emotions more accurately.

[1441] server

[1442] Based on the generated summary text and sentiment data, the server suggests appropriate actions for the user. These suggestions are tailored to the user's emotional state. For example, if the user is stressed, suggestions that promote relaxation can be offered. Finally, the summary text and suggested actions are sent to the user's device.

[1443] User

[1444] Users can view summary text and suggested actions through their device screen. For example, if someone nearby suggests going to a cafe, the device will display the summary text "Suggestion to go to a cafe" and the suggested action "Next action: Go to a cafe." Furthermore, if the user needs to relax, a suggestion such as "How about going to a cafe to relax?" will be made. Users can then act accordingly.

[1445] Specific example

[1446] Conversation with someone nearby

[1447] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[1448] 2. Terminal: Sends the acquired audio data to the server.

[1449] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "Shall we go for coffee at a nearby cafe?"

[1450] 4. Server: Uses a generative AI to summarize the text data and generate the phrase "A suggestion to go to a cafe."

[1451] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is experiencing stress.

[1452] 6. Server: Based on emotional data, it generates a response such as "Next action: Go to a cafe, but spend the time relaxing."

[1453] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[1454] 8. Terminal: Display summary text and countermeasures on the user's screen.

[1455] 9. User: Review the displayed information and decide on the next course of action.

[1456] Public announcements at train stations

[1457] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[1458] 2. Terminal: Sends the acquired audio data to the server.

[1459] 3. Server: Receives the voice data and uses a speech recognition engine to convert it into text, "The train is delayed. The next train will arrive in 20 minutes."

[1460] 4. Server: Uses a generative AI to summarize text data and generate the message, "The train is delayed. The next one will arrive in 20 minutes."

[1461] 5. Server: Analyzes the user's facial expressions and tone of voice to recognize if the user is anxious.

[1462] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[1463] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[1464] 8. Terminal: Display summary text and countermeasures on the user's screen.

[1465] 9. User: Review the displayed information and decide on the next course of action.

[1466] Thus, this system enables people with hearing impairments to understand surrounding auditory information in real time and to respond appropriately according to their emotions.

[1467] The following describes the processing flow.

[1468] Program processing details

[1469] Step 1:

[1470] The device uses a microphone to capture ambient sounds around the user. The microphone is always ON, collecting audio data in real time.

[1471] Step 2:

[1472] The device stores the acquired audio data in temporary memory. The audio data is then divided into packets at regular intervals.

[1473] Step 3:

[1474] The terminal packets the voice data and sends it to the server using wireless communication (Wi-Fi, 4G, 5G, etc.). The transmission is optimized for low latency.

[1475] Step 4:

[1476] The server reconstructs the received audio data and saves it as an audio file. This makes subsequent processing easier.

[1477] Step 5:

[1478] The server starts the speech recognition engine and analyzes the stored audio file. The audio data is then converted into text data.

[1479] Step 6:

[1480] The server inputs the converted text data into a natural language processing (NLP) engine for analysis. This analysis includes understanding the context and extracting key points.

[1481] Step 7:

[1482] The server uses generative AI to generate a summary from text data. The summary is presented in a concise form that includes the main information.

[1483] Step 8:

[1484] The server activates the emotion engine to analyze the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotions from the specified input data.

[1485] Step 9:

[1486] The server analyzes emotional data to determine, for example, levels of stress and relaxation. This process also utilizes the learning results from past emotional data.

[1487] Step 10:

[1488] The server proposes appropriate actions for the user based on the generated summary text and sentiment data. These proposals are adjusted according to the user's current emotional state.

[1489] Step 11:

[1490] The server converts the generated summary text and countermeasures into data packets and sends them to the terminal.

[1491] Step 12:

[1492] The terminal decodes the received data packets and converts them into a format for display on the user interface.

[1493] Step 13:

[1494] The device displays a summary text and countermeasures on the user's screen. The display is visually easy to understand, and a notification function is also included for emergencies.

[1495] Step 14:

[1496] The user checks the device screen and decides on the next action based on the displayed summary text and countermeasures.

[1497] Specific example

[1498] Conversation with someone nearby

[1499] 1. Step 1: The device uses its microphone to pick up the voice of someone nearby saying, "Want to go get some coffee at a nearby cafe?"

[1500] 2. Step 2: The device saves the acquired audio data to temporary memory and then packets it.

[1501] 3. Step 3: The terminal sends the packetized voice data to the server using wireless communication.

[1502] 4. Step 4: The server reconstructs the received audio data and saves it as an audio file.

[1503] 5. Step 5: The server activates the speech recognition engine to analyze the saved audio file and convert it into text data.

[1504] 6. Step 6: The server inputs the text data into the natural language processing engine for analysis. Analysis includes understanding the context and extracting key points.

[1505] 7. Step 7: The server uses generative AI to generate a summary. It generates the suggestion "Let's go to a cafe."

[1506] 8. Step 8: The server analyzes the user's facial expressions and voice tone using an emotion engine.

[1507] 9. Step 9: The server analyzes the emotional data and determines that the user needs to relax.

[1508] 10. Step 10: Based on the summary text "Suggestion to go to a cafe," the server generates a response that reflects sentiment data: "Next action: Go to a cafe. Spend time relaxing."

[1509] 11. Step 11: The server packets the summary text and countermeasures and sends them to the terminal.

[1510] 12. Step 12: The terminal decrypts the received data and saves it to local memory.

[1511] 13. Step 13: The device displays a summary text and a solution on the screen.

[1512] 14. Step 14: The user reviews the information on the screen and decides on their next action.

[1513] (Example 2)

[1514] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1515] Traditional systems designed to facilitate communication for the hearing impaired in urban environments have suffered from problems such as low accuracy in recognizing ambient sounds and converting text, and a lack of appropriate responses based on emotional states. Furthermore, many systems struggled to provide real-time solutions and adequately captured changes in the user's emotions. This resulted in situations where the hearing impaired were unable to take appropriate action.

[1516] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1517] In this invention, the server includes means for acquiring ambient sounds with an audio input device, means for transmitting the acquired audio data to an information processing device, means for converting the audio data into language data, means for analyzing the language data and generating a summary, means for proposing countermeasures based on the summary text and the user's emotional state, means for notifying the user of the summary text and countermeasures, and means for analyzing the user's emotional state. This enables hearing-impaired individuals to recognize ambient sounds in real time and respond appropriately according to their emotions.

[1518] "Environmental sounds" refer to sounds that occur in everyday life, such as the voices of people around you or public broadcasts.

[1519] A "voice input device" is a device that includes a microphone or other voice input means for acquiring voice data.

[1520] An "information processing device" refers to a computer or server with data processing capabilities, and is a device used to process acquired audio data.

[1521] "Linguistic data" refers to text data and linguistic information obtained by analyzing audio data.

[1522] "Analysis" is the process of understanding the meaning and context based on acquired data, and then further processing the understood information.

[1523] "Summarization" refers to the process of extracting important content and key points from analyzed information in a concise manner.

[1524] "Generation" refers to creating new data or information based on given data.

[1525] "Recommendations" refer to suggesting actions or countermeasures to users based on analysis results and generated data.

[1526] A "notification" is a means of informing the user of the generated summary text and proposed countermeasures.

[1527] "Users" refers to hearing-impaired individuals and other users who use this system.

[1528] "Emotional state" refers to the user's current psychological situation and emotions, including states such as stress, satisfaction, and anxiety.

[1529] An "emotion engine" refers to software or hardware that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1530] An "artificial intelligence model" refers to artificial intelligence technology that learns from vast amounts of data to make predictions and generate data, and specifically includes generative AI.

[1531] This invention is a system designed to facilitate communication for people with hearing impairments in urban areas. The system is realized by combining a voice input device, an information processing device, a speech recognition engine, a generative AI model, and an emotion engine.

[1532] First, a smartphone or dedicated device is used as the terminal. The terminal has a built-in microphone that captures ambient sounds in real time. For example, the terminal records a conversation such as "Shall we go for coffee at a nearby cafe?" using its microphone. This captured audio data is transmitted to a server via wireless communication such as Wi-Fi, 4G, or 5G.

[1533] The server passes the received audio data to a speech recognition engine (for example, general speech recognition software), which converts the audio data into text data. For example, the audio data "Want to go get some coffee at a nearby cafe?" is converted into text data.

[1534] Next, the text data is analyzed by a generative AI model (e.g., a well-known generative AI model) to summarize the important information. In this process, the generative AI performs appropriate contextual analysis on the input text data and generates a summary such as "a suggestion to go to a cafe."

[1535] Furthermore, the server is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice to understand their emotional state. The emotion engine also refers to past emotional data to more accurately recognize the user's current emotions. For example, if the system detects that the user is feeling stressed, "stress" will be recorded as emotional data.

[1536] The server generates appropriate countermeasures based on this sentiment data and the generated summary text. Using a generative AI model, it suggests countermeasures tailored to the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing."

[1537] Finally, the generated summary text and suggested actions are sent back to the device via wireless communication. The user's device receives this and notifies the user by displaying it on the screen. For example, the device's display might show actions such as "Go to a cafe" and "Spend some time relaxing."

[1538] The user reviews the displayed information and decides on their next action based on it. For example, the user decides whether to follow the suggestion to "go to a cafe" and then takes action.

[1539] Specific example

[1540] Example 1: Conversation with someone nearby

[1541] 1. Device: The microphone captures the voice of a nearby person saying, "Want to go get some coffee at a nearby cafe?"

[1542] 2. Terminal: Sends the acquired audio data to the server. Uses wireless communication such as Wi-Fi.

[1543] 3. Server: Converts audio data into text data using a speech recognition engine.

[1544] 4. Server: Analyzes text data using a generative AI model and summarizes it as "a suggestion to go to a cafe."

[1545] 5. Server: The emotion engine analyzes the user's emotional state and recognizes that the user is experiencing stress.

[1546] 6. Server: Based on sentiment data and summary text, it generates a response: "Next action: Go to a cafe, but spend the time relaxing."

[1547] 7. Server: Sends a summary text and countermeasures to the user's terminal.

[1548] 8. Terminal: Display the summary text and countermeasures on the screen.

[1549] 9. User: Review the displayed information and decide on the next course of action.

[1550] Example 2: Public announcements at train stations

[1551] 1. Terminal: The device uses a microphone to capture the audio of a public announcement at the station, such as "The train is delayed. The next train will arrive in 20 minutes."

[1552] 2. Terminal: Sends the acquired audio data to the server.

[1553] 3. Server: The speech data is converted into text, "The train is delayed. The next train will arrive in 20 minutes," using a speech recognition engine.

[1554] 4. Server: A generative AI model analyzes the text data and summarizes it as, "The train is delayed. The next one will arrive in 20 minutes."

[1555] 5. Server: Uses an emotion engine to analyze the user's emotional state and recognizes that the user is feeling anxious.

[1556] 6. Server: Based on emotional data, it generates countermeasures such as "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[1557] 7. Server: Sends summary text and countermeasure data to the user's terminal.

[1558] 8. Terminal: Display summary text and countermeasures on the user's screen.

[1559] 9. User: Review the displayed information and decide on the next course of action.

[1560] In this way, this system supports people with hearing impairments in understanding surrounding sounds in real time and taking appropriate responses based on their emotions.

[1561] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1562] Step 1:

[1563] Terminal: The system acquires ambient sounds using an audio input device (microphone). A microphone built into the user's smartphone or dedicated device captures ambient sounds and converts them into digital audio data. The input is analog ambient sound, and the output is digital audio data.

[1564] Step 2:

[1565] Terminal: Transmits acquired voice data to the information processing device (server) via wireless communication (Wi-Fi, 4G, 5G, etc.). The communication module compresses the voice data and sends it to the server's API endpoint using the appropriate communication protocol. Input is digital voice data, and output is data transmission to the server.

[1566] Step 3:

[1567] Server: Receives audio data sent from the terminal and passes it to the speech recognition engine, converting it into language data (text data). The speech recognition engine analyzes the audio data and generates the corresponding text data. For example, it converts the audio data "Shall we go for coffee at a nearby cafe?" into the text data "Shall we go for coffee at a nearby cafe?". The input is digital audio data, and the output is text data.

[1568] Step 4:

[1569] Server: Uses a generative AI model to analyze text data and summarize important information based on context. The AI ​​model understands the intent and context of the text and generates summarized text such as "a suggestion to go to a cafe." Input is text data, output is summarized text.

[1570] Step 5:

[1571] Server: Uses an emotion engine to analyze the user's facial expressions and voice tone transmitted from the terminal to understand their emotional state. Image data and audio data captured by the user's camera are analyzed to derive emotional states such as stress, satisfaction, and anxiety. Input is image data and audio data, and output is emotion data.

[1572] Step 6:

[1573] Server: Generates appropriate countermeasures based on sentiment data and generated summary text. Using a generative AI model, it suggests the most suitable countermeasure for the user's emotional state, such as "Next action: Go to a cafe, but spend the time relaxing." Input is sentiment data and summary text, output is the countermeasure.

[1574] Step 7:

[1575] Server: Sends the generated summary text and countermeasures back to the user's terminal via wireless communication. The data transmission module compresses the summary text and countermeasures and sends the data to the terminal's API endpoint. The input is the summary text and countermeasures, and the output is data transmission to the terminal.

[1576] Step 8:

[1577] Terminal: Displays the received summary text and countermeasures on the user's screen. The user interface visually displays the text data and notifies the user. For example, the display might show "Let's go to a cafe" and "Spend time relaxing." The input is the summary text and countermeasures, and the output is the display on the screen.

[1578] Step 9:

[1579] User: Review the displayed summary text and suggested actions to determine the next course of action. The user takes a specific action based on the information provided, such as deciding to go to a cafe. The input is the information displayed on the screen, and the output is the user's action.

[1580] In this way, the system allows hearing-impaired individuals to understand surrounding sounds in real time and suggest appropriate responses based on their emotions.

[1581] (Application Example 2)

[1582] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1583] This project aims to address the problem of hearing-impaired individuals having difficulty understanding the speech of people nearby or public broadcasts in public spaces, and the lack of means to appropriately manage the user's emotions and provide appropriate support in such situations.

[1584] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting audio data into text, means for analyzing text data and generating a summary, and means for recognizing the user's emotions and adjusting countermeasures based on acquired emotion data. This enables hearing-impaired individuals to understand surrounding audio information in real time and to respond appropriately according to their emotions.

[1585] "Ambient sounds" refer to all kinds of sounds that occur around the user.

[1586] A "microphone" is a device used to convert sound into electrical signals.

[1587] "Audio data" refers to data that represents audio captured by a microphone in digital format.

[1588] A "server" is a computer device that receives voice data via a communication network and performs analysis processing.

[1589] "Text" refers to audio data converted into written form.

[1590] "Generative AI" refers to artificial intelligence systems that analyze given text data and generate summaries or suggestions.

[1591] "Emotional data" refers to data that indicates the user's emotional state, obtained through the user's facial expressions, tone of voice, and other biosensors.

[1592] "Countermeasures" refer to the next actions or appropriate responses suggested to the user based on text data and sentiment data.

[1593] A "means of notification" refers to a system for visually and audibly conveying summary text and countermeasures to the user.

[1594] This invention is a system that makes it easier for hearing-impaired individuals to understand the speech of people nearby and public broadcasts in urban environments, and further recognizes the user's emotions and adjusts the response accordingly. The following describes a specific embodiment of this system.

[1595] To use the system, users wear smart glasses. These smart glasses have a built-in microphone and camera, which can capture ambient sounds and the user's facial expressions in real time.

[1596] Hardware and software to be used

[1597] Smart glasses: Generally, Google Glass or Vuzix Blade are used.

[1598] Server: Uses a high-performance computing system to perform speech recognition, generative AI, and emotion recognition.

[1599] Speech recognition engine: Uses Google Cloud Speech-to-Text.

[1600] Generative AI: Utilizes OpenAI GPT.

[1601] Emotion engine: Utilizes the Microsoft Azure Emotion API.

[1602] System Operation Overview

[1603] 1. Audio acquisition

[1604] The microphone in the smart glasses, which is the device itself, captures ambient sounds in real time. This audio data is transmitted to a server using wireless communication such as Bluetooth or Wi-Fi.

[1605] 2. Speech Recognition

[1606] The server converts the received audio data into text data using a speech recognition engine (Google Cloud Speech-to-Text). This conversion allows the spoken content to be treated as text information.

[1607] 3. Text Analysis and Summarization

[1608] The converted text data is passed to a generation AI (OpenAI GPT) for contextual analysis and summarization. This extracts the most important information from the spoken content and generates a short summary.

[1609] 4. Emotion recognition

[1610] The server receives user facial expression data captured using the smart glasses' camera and retrieves emotion data using the emotion engine (Microsoft Azure Emotion API). This allows the server to evaluate the user's current emotional state.

[1611] 5. Proposed countermeasures

[1612] The server uses generative AI to suggest countermeasures based on summarized text and sentiment data. Specific action suggestions are tailored depending on whether the user is experiencing stress or needs relaxation.

[1613] 6. Notification to the user

[1614] The final summary text and suggested countermeasures are displayed on the smart glasses' screen. This allows users to understand surrounding speech and public announcements in real time and take action according to their own emotional state.

[1615] Specific example

[1616] Conversation with someone nearby

[1617] 1. The server receives the utterance "Want to go for coffee at a nearby cafe?" picked up by the microphone on the smart glasses from a nearby person.

[1618] 2. Convert the audio data into text and use a generation AI to summarize it as "a suggestion to go to a cafe."

[1619] 3. If the user's facial expression indicates they are experiencing stress, a suggestion such as "How about going to a cafe to relax?" will be displayed.

[1620] Public announcements at train stations

[1621] 1. The smart glasses' microphone picks up the announcement, "The train is delayed. The next train will arrive in 20 minutes," and sends it to the server.

[1622] 2. Convert the audio data into text and summarize it as, "The train is delayed. The next one will arrive in 20 minutes."

[1623] 3. If the user is anxious, suggest a course of action such as, "Next action: Take the train in 20 minutes, but wait calmly in the meantime."

[1624] Example of a prompt

[1625] Summarize the following statement: "Want to go get some coffee at a nearby cafe?" Summary:

[1626] Suggestion when the user is sad: Suggest going to a cafe. Next action:

[1627] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1628] Step 1:

[1629] The device uses a microphone to acquire ambient sounds in real time. These ambient sounds include conversations and public broadcasts. The acquired audio data is converted into a digital format and transmitted to a server via wireless communication (Wi-Fi or Bluetooth).

[1630] Input: Ambient sounds

[1631] Output: Digital audio data

[1632] Step 2:

[1633] The server converts the received audio data into text using a speech recognition engine (Google Cloud Speech-to-Text). The speech recognition engine analyzes the audio data and represents it as textual information.

[1634] Input: Digital audio data

[1635] Output: Converted text data

[1636] Step 3:

[1637] The server uses a generative AI (OpenAI GPT) to analyze text data and generate a summary. The generative AI performs contextual analysis and extracts important information to generate the summarized text.

[1638] Input: Converted text data

[1639] Output: Summarized text data

[1640] Step 4:

[1641] The device's camera is used to acquire user facial expression data. This data is used to recognize the user's emotions. The acquired facial expression data is transmitted to a server via wireless communication.

[1642] Input: User facial expression data

[1643] Output: Digital facial expression data

[1644] Step 5:

[1645] The server uses an emotion engine (Microsoft Azure Emotion API) to analyze the user's facial expressions and obtain emotional data. The emotion engine performs facial analysis and outputs the user's emotional state as data.

[1646] Input: Digital facial expression data

[1647] Output: Sentiment data

[1648] Step 6:

[1649] The server uses generative AI to suggest appropriate responses based on the summarized text and sentiment data. These responses are adjusted according to the user's emotional state. For example, if the user is feeling stressed, suggestions for relaxation will be offered.

[1650] Input: Summary text, sentiment data

[1651] Output: Proposed countermeasures

[1652] Step 7:

[1653] The device receives the summarized text and countermeasures sent from the server and displays them on the smart glasses' screen. This allows the user to see the summarized information and suggestions in real time and decide on their next course of action.

[1654] Input: Summary text, proposed countermeasures

[1655] Output: Summary information and suggestions displayed on the screen.

[1656] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1657] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1658] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1659] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1660] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1661] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1662] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1663] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1664] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1665] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1666] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1667] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1668] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1669] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1670] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1671] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1672] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1673] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1674] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1675] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1676] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1677] The following is further disclosed regarding the embodiments described above.

[1678] (Claim 1)

[1679] Methods for capturing ambient sounds with a microphone,

[1680] A means of sending the acquired audio data to the server,

[1681] A means of converting audio data into text,

[1682] A means for analyzing text data and generating a summary,

[1683] A means of proposing countermeasures based on the summarized text,

[1684] A system that includes means for notifying the user of summary text and countermeasures.

[1685] (Claim 2)

[1686] The system according to claim 1, which acquires and transmits ambient sounds by wireless communication.

[1687] (Claim 3)

[1688] The system according to claim 1, which uses a generative AI to propose summary text.

[1689] "Example 1"

[1690] (Claim 1)

[1691] A method for acquiring ambient sounds using a microphone,

[1692] A means of sending the acquired acoustic data to the server,

[1693] A means of converting audio data into text,

[1694] A means for analyzing text data and generating a summary,

[1695] A means of proposing a response based on the generated summary text,

[1696] A system that includes means for notifying the user of summary text and response measures.

[1697] (Claim 2)

[1698] The system according to claim 1, which acquires and transmits ambient sounds wirelessly.

[1699] (Claim 3)

[1700] The system according to claim 1, which uses generative artificial intelligence to generate summary text and propose response strategies.

[1701] "Application Example 1"

[1702] (Claim 1)

[1703] Means for acquiring ambient sounds,

[1704] A means of sending the acquired audio data to a remote server,

[1705] A means of converting audio data into text,

[1706] A means for analyzing text data and generating a summary,

[1707] A means of proposing countermeasures based on the summarized text,

[1708] A system including means for notifying a display device of a summary text containing countermeasures.

[1709] (Claim 2)

[1710] The system according to claim 1, which acquires ambient sounds and transmits voice data wirelessly.

[1711] (Claim 3)

[1712] The system according to claim 1, which uses generative artificial intelligence to propose countermeasures.

[1713] "Example 2 of combining an emotion engine"

[1714] (Claim 1)

[1715] A means of acquiring ambient sounds using an audio input device,

[1716] A means for transmitting acquired audio data to an information processing device,

[1717] A means of converting audio data into language data,

[1718] A means for analyzing linguistic data and generating summaries,

[1719] A means of proposing countermeasures based on the summarized text and the user's emotional state,

[1720] A means of notifying users of the summary text and countermeasures,

[1721] A system that includes means for analyzing the emotional state of users.

[1722] (Claim 2)

[1723] The system according to claim 1, which acquires and transmits ambient sounds by wireless communication.

[1724] (Claim 3)

[1725] The system according to claim 1, which uses an artificial intelligence model to generate summary text.

[1726] (Claim 4)

[1727] The system according to claim 1, which uses past emotional data to analyze the emotions of users.

[1728] "Application example 2 when combining with an emotional engine"

[1729] (Claim 1)

[1730] Methods for capturing ambient sounds with a microphone,

[1731] A means of sending the acquired audio data to the server,

[1732] A means of converting audio data into text,

[1733] A means for analyzing text data and generating a summary,

[1734] A means of proposing countermeasures based on the summarized text,

[1735] A means of recognizing user emotions and adjusting countermeasures based on acquired emotional data,

[1736] A system that includes means for notifying the user of summary text and countermeasures.

[1737] (Claim 2)

[1738] The system according to claim 1, which acquires and transmits ambient sounds by wireless communication.

[1739] (Claim 3)

[1740] The system according to claim 1, which uses a generative AI to propose summary text. [Explanation of symbols]

[1741] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Methods for capturing ambient sounds with a microphone, A means of sending the acquired audio data to the server, A means of converting audio data into text, A means for analyzing text data and generating a summary, A means of proposing countermeasures based on the summarized text, A system that includes means for notifying the user of summary text and countermeasures.

2. The system according to claim 1, which acquires and transmits ambient sounds by wireless communication.

3. The system according to claim 1, which uses a generative AI to propose summary text.

Images