system

Abstract

We provide the system. [Solution] An acquisition means for acquiring audio signals, Amplification means for amplifying the aforementioned audio signal, A conversion means that converts the amplified audio signal into text in real time, Display means for displaying the converted text, Support measures to facilitate dialogue between care workers and the elderly, A system that includes this.

Description

Technical Field

[0004] , ,

[0005] , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Elderly people with reduced hearing or people with hearing impairments have difficulty communicating with others in their daily lives, which may lead to feelings of isolation and a decline in the quality of life. In particular, in public places or noisy environments, the clarity of speech is impaired, making conversations even more difficult to hold. There is a need to solve such problems and reduce communication barriers.

Means for Solving the Problems

[0005] This invention provides a system that acquires audio signals, amplifies them in real time to provide clear audio, and further converts that audio into text for visual presentation. Specifically, it is a system that combines an acquisition means for acquiring audio signals, an amplification means for amplifying them, a conversion means for converting them, and a display means for displaying them. Furthermore, it supports smooth communication at all times by notifying the user when an anomaly is detected. In addition, by processing the audio signal while removing noise, it enables stable use even in noisy environments.

[0006] A "speech signal" is an electrical signal representing sound waves emitted from the vocal cords, and it includes linguistic information and acoustic characteristics.

[0007] "Acquisition means" refers to a method or apparatus for capturing audio signals using a device such as a microphone.

[0008] "Amplification means" refers to a method or apparatus for increasing the amplitude of an acquired audio signal so that the audio can be heard more clearly.

[0009] "Conversion means" refers to the process or technology of converting audio signals into text data, and often uses speech recognition technology.

[0010] "Display means" refers to a method or apparatus for visually presenting converted text data, and includes screens and displays.

[0011] "Anomaly" refers to errors or malfunctions that disrupt normal operation in the acquisition or processing of audio signals.

[0012] "Noise" refers to unwanted acoustic components contained in an audio signal, which interfere with the intended audio information. [Brief explanation of the drawing]

[0013] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention provides a system for acquiring audio signals, amplifying them in real time, and transcribing them into text. Here, the program's processing for effectively handling audio signals is described in natural language.

[0035] First, the user installs and launches the application on a device such as a smartphone or tablet. To begin voice input, the device uses its built-in microphone to capture ambient sounds. At this point, noise cancellation technology is used to reduce background noise and process the sound to obtain clear audio.

[0036] Next, the audio signal acquired by the device is amplified in real time and output to the user's earphones or speakers to improve sound quality. The noise-reduced audio signal is then sent to a server in the cloud for speech recognition.

[0037] The server uses advanced speech recognition technology and natural language processing to convert received audio signals into text data. This converted text data is sent back to the terminal in real time, where the terminal visually displays the text. Here, the font size and color are adjusted to make it easy for elderly people to read. By providing information through both sight and sound, the quality of communication is improved.

[0038] As a concrete example, consider a student attending a lecture. This student can listen to the lecturer's explanation in audio format and simultaneously visualize the content as text. In this case, the text is updated in real time and displayed in sync with the audio, allowing the student to easily understand the lecture content and take notes.

[0039] If the terminal detects an abnormality or error, it can display an error message to the user and prompt them to re-enter the information, thus maintaining smooth communication at all times. In this way, the invention not only improves the quality of life for elderly people with hearing loss and those with hearing impairments, but is also expected to have applications in various environments.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The user launches an application on their device and operates the interface to initiate voice input. This starts the collection of the voice signal.

[0043] Step 2:

[0044] The device uses its built-in microphone to acquire audio signals in real time. Noise cancellation technology is then applied to process the signal to obtain an acoustically clear signal.

[0045] Step 3:

[0046] The device amplifies the audio signal it receives using a specified algorithm, adjusts the volume, and then outputs it to the user's earphones or speaker.

[0047] Step 4:

[0048] The device sends the noise-reduced audio signal to a server in the cloud for speech recognition processing. The transmission is performed using a secure communication protocol.

[0049] Step 5:

[0050] The server converts the received audio signal into text data using speech recognition technology. If necessary, natural language processing is applied to analyze the sentence structure and generate accurate text.

[0051] Step 6:

[0052] The server sends the converted text back to the terminal in real time. Optimization is performed here to minimize communication delays.

[0053] Step 7:

[0054] The device displays received text on the screen. The display format is customizable for user readability, supporting adjustments to font size, background color, and other settings.

[0055] Step 8:

[0056] Users can adjust volume and text display settings within the application as needed, optimizing them for their individual needs.

[0057] Step 9:

[0058] If the device detects a system anomaly, it immediately notifies the user and guides them to try voice input again if necessary. Seamless error handling during operation maintains system stability.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] There is a need to convert audio data into text data in real time and to quickly notify users when an anomaly is detected. In particular, the challenge is to accurately process information while effectively reducing background noise and improving sound quality, thereby enhancing the quality of life.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for transmitting audio data, means for analyzing the audio data on the cloud and converting it into text data, and means for receiving the text data and displaying it visually. This combines advanced background noise reduction technology with a process for improving sound quality, enabling accurate real-time information provision and rapid notification in the event of anomalies.

[0064] "Audio data" refers to data extracted from audio signals and represented in digital format.

[0065] "Device" refers to a combination of hardware and software for acquiring, processing, or displaying audio data.

[0066] "Background noise reduction" refers to the process of removing unwanted noise from audio data and emphasizing the main audio portion.

[0067] "Amplifying" is the process of increasing the signal strength of audio data and improving its clarity.

[0068] "Transmission" refers to the act of transferring processed audio data to another device or cloud server.

[0069] "Analysis on the cloud" refers to analyzing audio data in a cloud computing environment and converting it into text data.

[0070] "Text data" refers to digital text converted from audio data and displayed in a visually recognizable format.

[0071] "Visual display" means presenting text data on a display in a format that is recognizable to the user.

[0072] "Detecting anomalies" refers to the process of discovering a state within a system that deviates from normal operation and notifying the user of that information.

[0073] "User" refers to an individual who uses the system to acquire or display voice data.

[0074] This invention is a system for acquiring audio data in real time, converting it into text data, and displaying it visually. Its primary purpose is to improve the quality of the audio data while reducing background noise. Users install an application on information terminals such as smartphones and tablets and utilize a cloud computing environment. Typically, the microphone built into the information terminal is used to acquire the audio data.

[0075] The device reduces background noise from the acquired audio data by applying noise cancellation technology. This process may utilize an audio processor incorporating existing acoustic technologies. The noise-free audio data is amplified in real time and output to the user. A secure protocol is used for transmitting the audio data over the internet.

[0076] The server analyzes the received audio data in the cloud. This analysis utilizes speech recognition technology and natural language processing algorithms, such as a common speech recognition API. The analyzed audio data is returned to the device as text data. On the device, this text data is displayed visually, and the font size and color settings can be adjusted according to the user's preferences.

[0077] Specific examples include students attending lectures and business people participating in meetings. By receiving text information in real time simultaneously with voice input, misunderstandings and missed information can be prevented, enabling efficient information processing. This system is particularly expected to be used by people with hearing impairments and the elderly.

[0078] An example of a prompt for a generative AI model is: "Please explain the process of a speech recognition system that transcribes lectures into text in real time." This allows the present invention to flexibly adapt to various use cases.

[0079] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0080] Step 1:

[0081] The user installs and launches the application on their smartphone or tablet.

[0082] The input is a user action (launching the app).

[0083] As an output, the device is set to voice input mode, and a "Start Voice Input" button is displayed on the interface.

[0084] Step 2:

[0085] The device receives a tap of the "Start Voice Input" button as input from the user and begins capturing ambient sounds with its built-in microphone.

[0086] The input includes ambient sounds acquired through the built-in microphone.

[0087] Using noise cancellation technology, background noise is reduced from the input audio, generating clean audio data.

[0088] The output is audio data with noise removed, and the process proceeds to the next processing step.

[0089] Step 3:

[0090] The device amplifies the noise-reduced audio data in real time.

[0091] The input is audio data after noise reduction.

[0092] As part of the data processing, signal amplification is performed to improve the clarity of the audio.

[0093] The output is amplified audio data, which is then output in real time to the user's earphones or speakers.

[0094] Step 4:

[0095] The device compresses the amplified audio data and sends it to a server in the cloud.

[0096] The input is amplified audio data.

[0097] As part of the data processing, the audio data is compressed to make it ready for transmission.

[0098] The compressed audio data is sent to the server as output.

[0099] Step 5:

[0100] The server analyzes the audio data it receives.

[0101] The input is audio data sent from the device.

[0102] For data processing, speech recognition technology and natural language processing are used to convert speech data into text data in real time.

[0103] As output, converted character data is generated and sent to the terminal.

[0104] Step 6:

[0105] The terminal visually displays the character data received from the server.

[0106] The input is character data returned from the server.

[0107] Data processing involves adjusting font size, color, and other settings to format the information as optimal visual data for the user.

[0108] As output, formatted text data is displayed on the terminal's screen, allowing the user to verify the information.

[0109] (Application Example 1)

[0110] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0111] It is difficult for elderly and hearing-impaired users to communicate effectively in care settings. There is a need for technology that removes communication barriers by providing not only audio information but also visual information in real time for these users.

[0112] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0113] In this invention, the server includes an acquisition means for acquiring an audio signal, an amplification means for amplifying the audio signal, and a conversion means for converting the amplified audio signal into text in real time. This enables smooth communication between caregivers and elderly individuals by converting audio information into visual information in real time.

[0114] An "audio signal" is an electrical signal that converts sound information transmitted by air vibrations into an electrical signal.

[0115] "Acquisition means" refers to a device or method for collecting audio signals.

[0116] "Amplification means" refers to a device or method for converting an audio signal to a higher volume.

[0117] "Conversion means" refers to a device or method for converting an audio signal into text data.

[0118] "Display means" refers to a device or method for visually displaying the converted text.

[0119] "Care workers" are people who are engaged in jobs that support the lives of elderly people.

[0120] "Elderly people" generally refers to people who are considered to be older.

[0121] "Support measures" refer to devices or methods that facilitate communication between care workers and elderly individuals.

[0122] "Noise" refers to unwanted sounds that are mixed in with the intended audio signal.

[0123] "Notification means" refers to a device or method for informing a user of an anomaly or event.

[0124] The system for carrying out this invention is configured to acquire, amplify, convert to text, and display audio signals in real time, thereby supporting communication between caregivers and elderly individuals. This system mainly includes audio acquisition means, amplification means, conversion means, and display means. The server uses advanced speech recognition technology and natural language processing to accurately convert the acquired audio signals into text data and transmit it to the user's terminal. The terminal displays this as visual information, allowing the user to confirm it both aurally and visually.

[0125] The primary hardware includes the user's smartphone or tablet, as well as earphones and speakers. The software utilizes noise cancellation technology (e.g., Active Noise Cancellation), speech recognition APIs (e.g., Google® Cloud Speech-to-Text), and natural language processing libraries (e.g., spaCy). These technologies, combined with the server's advanced processing of audio signals, enable users to receive audio as visual information in real time.

[0126] As a concrete example, consider a scenario in a nursing home where caregivers give instructions regarding meals and medication to elderly residents. The caregiver's voice is amplified and converted into text on the resident's device, and displayed as visual information on the screen, making it easier for the resident to accurately understand the instructions.

[0127] An example of a prompt for a generative AI model might be: "Imagine a scenario in a nursing home where care staff are giving instructions to residents. Explain how a real-time speech conversion app can support smooth communication." This prompt would prompt the generative AI model to describe specific support measures and their effects.

[0128] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0129] Step 1:

[0130] The device uses its built-in microphone to acquire ambient audio signals. The acquired audio signal is then used as input, and background noise is removed using noise-canceling technology. This allows for the output of a clear audio signal.

[0131] Step 2:

[0132] The device amplifies the noise-reduced audio signal and outputs it to the user's earphones or speakers. The audio signal is amplified in real time and provided at a volume that is easier for the user to hear.

[0133] Step 3:

[0134] The terminal sends the amplified audio signal to a server in the cloud. The server receives the audio signal as input and converts it into digital text using advanced speech recognition technology. The converted text data is then generated as output data.

[0135] Step 4:

[0136] The server sends the generated text data back to the terminal. The terminal takes the text data as input and displays it visually on the screen. The font size and color are adjusted so that the user can easily read it.

[0137] Step 5:

[0138] Users can confirm instructions from caregivers through displayed text. This allows for communication using both visual and auditory means.

[0139] Step 6:

[0140] The terminal monitors for system anomalies and displays an error message to the user if an anomaly is detected. This allows the user to obtain information to take the next action.

[0141] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0142] This invention combines a system that acquires user voice signals, amplifies them in real time, and transcribes them into text with an emotion engine. By using the emotion engine, it is possible to identify emotions from user speech and enable appropriate interaction.

[0143] The user installs and launches the application on a device such as a smartphone or tablet. This application has the function of acquiring audio signals through the microphone, and the acquired audio signals are processed in real time using noise cancellation technology. The device then amplifies these audio signals, providing the user with clear audio.

[0144] The audio signal is then sent to a server in the cloud, where it is converted into text using speech recognition and natural language processing technologies. During this process, an emotion engine analyzes the user's emotional state from the audio and extracts emotion data. This data indicates the user's current emotional state, and the analysis results are sent to the device.

[0145] The device dynamically adjusts the format of the text it displays based on acquired emotional data, for example, by changing the color and font size according to the user's emotions. Furthermore, when the device detects user stress or anxiety, it can automatically provide voice guidance for relaxation.

[0146] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can hear clear audio through the headset while reducing ambient noise, and simultaneously receive visual feedback in the form of text. Furthermore, if the emotion engine detects that the user is experiencing stress, the device can suggest playing relaxation music. In this way, the present invention provides a system that reduces the user's mental burden and supports smooth communication.

[0147] The following describes the processing flow.

[0148] Step 1:

[0149] The user launches the application on their device and selects voice input mode. This puts the device into a state where it can begin capturing audio.

[0150] Step 2:

[0151] The device uses its built-in microphone to acquire audio signals in real time. During this process, noise cancellation is enabled to remove ambient noise and obtain clear audio data.

[0152] Step 3:

[0153] The device analyzes the acquired audio signal and amplifies it to an appropriate volume for output. The amplified audio is then delivered to the user through earphones or speakers.

[0154] Step 4:

[0155] The device sends the de-noised audio signal to a server in the cloud, requesting data processing for speech recognition and sentiment analysis.

[0156] Step 5:

[0157] The server converts the received audio data into text using speech recognition technology. Simultaneously, it uses an emotion engine to analyze the user's emotional state from their voice and generates the results.

[0158] Step 6:

[0159] The server sends the analyzed text data and sentiment data back to the terminal. Communication is performed with low latency, prioritizing user convenience.

[0160] Step 7:

[0161] When a device displays received text on the screen, it automatically adjusts the formatting based on sentiment data. This includes changing the text color and font size according to the sentiment.

[0162] Step 8:

[0163] If the device detects negative emotions such as stress or anxiety, it will automatically provide voice guidance or music for relaxation. For example, it can play soothing music to help alleviate mental stress.

[0164] Step 9:

[0165] Users can adjust various settings options provided on their device as needed to enjoy a personalized experience. Other background sounds and responses that respond to specific emotions can also be configured.

[0166] (Example 2)

[0167] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0168] In modern society, users are required to handle audio information in various environments and understand the speaker's emotions within those contexts. However, conventional systems are susceptible to noise, making it difficult to accurately analyze emotional states. Furthermore, they lacked the functionality to provide appropriate feedback in response to the user's emotions. This invention aims to solve these problems and realize emotion-responsive interaction.

[0169] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0170] In this invention, the server includes means for acquiring audio information, means for amplifying the audio information, means for converting the amplified audio information into text information in real time, means for analyzing emotional information, and means for dynamically adjusting the display format based on the emotional information. This makes it possible to accurately analyze the emotional state of the user's speech and provide appropriate feedback while removing noise from the audio signal.

[0171] "Audio information" refers to sound waves represented as digital or analog signals.

[0172] "Means" refers to the equipment, apparatus, or method used to achieve a particular purpose.

[0173] "Amplification" refers to the process of increasing the strength or magnitude of a signal, making it clearer.

[0174] "Real-time" refers to a process that proceeds almost instantly, with virtually no time delay.

[0175] "Textual information" refers to a set of symbols and characters used to visually represent language.

[0176] "Display format" refers to the design and style that indicates how information and data are visually arranged.

[0177] "Dynamic adjustment" refers to the automatic modification or correction of information in response to changes in circumstances or conditions.

[0178] "Analysis" is the process of breaking down complex information or data into an easily understandable form and revealing its meaning and structure.

[0179] This invention is a system that enhances user interaction by acquiring audio information, amplifying it, converting it into text information, and further performing emotion analysis. Specifically, it is implemented as follows.

[0180] The user installs a dedicated application on their mobile device and starts the system by launching this application. This application uses the device's microphone to acquire voice information and utilizes noise cancellation technology to eliminate ambient noise, thus maintaining a clear voice signal.

[0181] Next, the device amplifies the acquired audio signal in real time, providing the user with clearer audio. This process ensures that the audio signal information is not lost, allowing the next step to be completed without any loss of quality.

[0182] The voice signal is transmitted via the internet to a server in the cloud. The server uses speech recognition technology to convert the voice signal into text information. Furthermore, a generative AI model is used to analyze and extract user emotional information from the voice. The information obtained from this emotional analysis is used to reflect and predict the user's psychological state.

[0183] The terminal dynamically adjusts the format of the displayed text information based on emotional information received from the server. For example, if the system determines that the user is calm, it can change the color scheme to a gentle blue, while if the user is excited, it can change to a red color scheme that indicates energy. This allows the system to provide a more user-friendly display.

[0184] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can use a headset to reduce ambient noise, hear clear audio, and simultaneously view visual text information on their device screen. Furthermore, if the user experiences stress, the system can provide relaxation guidance, creating a pleasant experience.

[0185] An example of a prompt statement would be, "I'm feeling stressed during my commute; please suggest some relaxation music." In this way, the present invention provides an effective method for understanding and promoting a user's emotions.

[0186] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0187] Step 1:

[0188] The user launches the application on their mobile device. As input, the user speaks into the device's microphone. The device acquires this audio signal and uses noise-canceling technology to remove unwanted ambient noise. The output is clear audio data.

[0189] Step 2:

[0190] Based on the acquired audio data, the device amplifies the audio signal, providing clear audio to the user in real time. The input is a noise-free audio signal, and the output is amplified audio. This amplification ensures that the user receives all the audio details clearly.

[0191] Step 3:

[0192] The terminal transmits amplified audio data to the server via the internet. The amplified audio signal, as input, becomes the data to be transmitted. The output is confirmation data after transmission is complete. This prepares the server to begin audio analysis.

[0193] Step 4:

[0194] The server uses speech recognition technology with the received audio data to convert speech into text. The input is the transmitted audio data, and the output is text data. The speech recognition engine records the spoken content as text.

[0195] Step 5:

[0196] The server uses a generative AI model to analyze user emotional information from text data. The input is text data, and the output is emotional information. The emotion engine identifies the user's psychological state and extracts the necessary emotional parameters.

[0197] Step 6:

[0198] Based on the emotional information, the server sends the analysis results to the terminal. The input is the analyzed emotional information, and the output is the emotional parameters sent to the terminal. This allows the terminal to prepare to display a response to the user.

[0199] Step 7:

[0200] The terminal dynamically adjusts the format of the text information displayed on the screen using the received emotion information. The input is emotion parameters received from the server, and the output is text information in the adjusted display format. Specifically, it changes the font color and size to display information according to the emotion.

[0201] Step 8:

[0202] Based on emotional information, the device detects user stress and anxiety and provides relaxation-enhancing audio guidance. Input is emotional information and the detection result of the user's stress level; output is playback of relaxation audio. This reduces the user's mental burden.

[0203] (Application Example 2)

[0204] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0205] A challenge exists in that when elderly people communicate with others, they may not receive appropriate responses that address their feelings, potentially increasing their stress and anxiety. While efficient and emotionally responsive support is needed in care settings, current technology has made it difficult to fully achieve this.

[0206] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0207] In this invention, the server includes a device for acquiring audio signals, a processing device for analyzing emotions, and a device for providing visual displays corresponding to the user's emotions. This makes it possible to analyze the emotions of elderly people and provide visual displays and audio guidance corresponding to those emotions.

[0208] A "speech signal" is a representation of sound as an electrical signal, and typically includes the content of human speech.

[0209] An "device" is a machine or system designed to achieve a specific function or purpose.

[0210] "Amplification" is the process of increasing the signal strength, and it is a process performed to make audio signals clearer.

[0211] "Real-time" refers to a state where processing is performed instantly without delay.

[0212] A "text" is a linguistic expression that uses words arranged according to certain rules to convey a specific meaning.

[0213] "Emotion" is a psychological state of a human being, and includes feelings such as joy and sadness.

[0214] "Analysis" is the process of breaking down an object into its constituent parts and understanding its properties.

[0215] A "processing device" is a device that manipulates or converts given data or signals according to instructions.

[0216] "Visual representation" means showing information in a way that is visible to the eye.

[0217] This invention is a system that allows users to receive real-time emotion analysis via voice signals and receive visual feedback and relaxation guidance. First, the terminal acquires the user's voice signal through a microphone. This terminal incorporates noise-canceling technology, which removes external noise and clarifies the voice signal. The voice signal is amplified and then transmitted to a cloud server.

[0218] When the server receives an audio signal, it uses speech recognition software to convert the signal into text. Next, the server uses an emotion analysis engine to identify the user's emotional state based on this text. This analysis may involve third-party speech recognition platforms or natural language processing technologies (e.g., OpenAI's GPT model).

[0219] The results of the emotion analysis are returned to the user's device, which then displays them visually. Furthermore, if stress or anxiety is detected, the device provides relaxation-enhancing audio guidance or music. This allows for appropriate responses tailored to the user's emotional state.

[0220] As a concrete example, consider a situation where an elderly person is conversing with a caregiver in a care facility. Using this system, the elderly person's voice is instantly converted into text, and their emotional state is analyzed. For example, if the emotional analysis determines that the elderly person is feeling anxious, the device can display a visually reassuring message and play relaxing music. This system allows caregivers to respond immediately to the elderly person's mental state.

[0221] An example of a prompt is, "Using this system, how would you like to support the emotional changes of elderly individuals during their daily communication?" Based on this prompt, the generative AI model can construct appropriate interactions that respond to their emotions.

[0222] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0223] Step 1:

[0224] The device acquires audio signals using a microphone. The input is the user's speech, and the output is digitized audio data. This data is processed into a clear audio signal by removing noise using noise cancellation technology.

[0225] Step 2:

[0226] The terminal sends a clear audio signal to the server. The input is a noise-free audio signal, and the output is the signal transfer to the cloud server. The server receives this signal and prepares for the next processing step.

[0227] Step 3:

[0228] The server uses a speech recognition engine to convert the audio signal into text. The input is the transmitted audio signal, and the output is the text data corresponding to that audio. This text is generated using technologies such as generative AI models.

[0229] Step 4:

[0230] The server uses an emotion analysis engine to identify the user's emotional state based on the converted text. The input is text data, and the output is emotion data. The server utilizes natural language processing techniques to analyze emotions.

[0231] Step 5:

[0232] The server instructs the system to generate visual feedback corresponding to the user's emotional state, and, if necessary, relaxation sounds. The input is emotional data, and the output is the instructed feedback content. Prompts may also be used in this process.

[0233] Step 6:

[0234] The terminal provides the user with visual displays and audio guidance based on feedback instructions from the server. The input is the instructions from the server, and the output is the user-viewable screen display and the audible relaxation audio.

[0235] Step 7:

[0236] The user follows the provided feedback, confirms the instructed actions, and performs relaxation as needed. Input is visual and auditory feedback, and output is the user's response or change in internal feelings.

[0237] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0238] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0239] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0240] [Second Embodiment]

[0241] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0242] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0243] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0244] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0245] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0246] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0247] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0248] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0249] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0250] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0251] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0252] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0253] This invention provides a system for acquiring audio signals, amplifying them in real time, and transcribing them into text. Here, the program's processing for effectively handling audio signals is described in natural language.

[0254] First, the user installs and launches the application on a device such as a smartphone or tablet. To begin voice input, the device uses its built-in microphone to capture ambient sounds. At this point, noise cancellation technology is used to reduce background noise and process the sound to obtain clear audio.

[0255] Next, the audio signal acquired by the device is amplified in real time and output to the user's earphones or speakers to improve sound quality. The noise-reduced audio signal is then sent to a server in the cloud for speech recognition.

[0256] The server uses advanced speech recognition technology and natural language processing to convert received audio signals into text data. This converted text data is sent back to the terminal in real time, where the terminal visually displays the text. Here, the font size and color are adjusted to make it easy for elderly people to read. By providing information through both sight and sound, the quality of communication is improved.

[0257] As a concrete example, consider a student attending a lecture. This student can listen to the lecturer's explanation in audio format and simultaneously visualize the content as text. In this case, the text is updated in real time and displayed in sync with the audio, allowing the student to easily understand the lecture content and take notes.

[0258] If the terminal detects an abnormality or error, it can display an error message to the user and prompt them to re-enter the information, thus maintaining smooth communication at all times. In this way, the invention not only improves the quality of life for elderly people with hearing loss and those with hearing impairments, but is also expected to have applications in various environments.

[0259] The following describes the processing flow.

[0260] Step 1:

[0261] The user launches an application on their device and operates the interface to initiate voice input. This starts the collection of the voice signal.

[0262] Step 2:

[0263] The device uses its built-in microphone to acquire audio signals in real time. Noise cancellation technology is then applied to process the signal to obtain an acoustically clear signal.

[0264] Step 3:

[0265] The device amplifies the audio signal it receives using a specified algorithm, adjusts the volume, and then outputs it to the user's earphones or speaker.

[0266] Step 4:

[0267] The device sends the noise-reduced audio signal to a server in the cloud for speech recognition processing. The transmission is performed using a secure communication protocol.

[0268] Step 5:

[0269] The server converts the received audio signal into text data using speech recognition technology. If necessary, natural language processing is applied to analyze the sentence structure and generate accurate text.

[0270] Step 6:

[0271] The server sends the converted text back to the terminal in real time. Optimization is performed here to minimize communication delays.

[0272] Step 7:

[0273] The device displays received text on the screen. The display format is customizable for user readability, supporting adjustments to font size, background color, and other settings.

[0274] Step 8:

[0275] Users can adjust volume and text display settings within the application as needed, optimizing them for their individual needs.

[0276] Step 9:

[0277] When the terminal detects an abnormality in the system, it immediately notifies the user and guides the user to try voice input again if necessary. By performing error handling during operation seamlessly, the stability of the system is maintained.

[0278] (Example 1)

[0279] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0280] It is required to convert voice data into character data in real time and quickly notify the user when an abnormality is detected. In particular, the problem is to accurately process information while effectively reducing background noise and improving the sound quality, and to improve the quality of life.

[0281] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0282] In this invention, the server includes means for transmitting voice data, means for analyzing voice data on the cloud and converting it into character data, and means for receiving character data and visually displaying it. As a result, by combining advanced background noise reduction technology and a process for improving sound quality, accurate information provision in real time and quick notification in case of an abnormality become possible.

[0283] "Voice data" is data extracted from a voice signal and expressed in digital form.

[0284] "Device" refers to a combination of hardware and software for acquiring, processing or displaying voice data.

[0285] "Reduction of background noise" refers to a process of removing unnecessary noise from voice data and emphasizing the main voice part.

[0286] "Amplifying" refers to the process of increasing the signal strength of audio data and improving clarity.

[0287] "Transmitting" refers to the act of transferring processed audio data to another device or a cloud server.

[0288] "Analyzing on the cloud" refers to analyzing audio data in a cloud computing environment and converting it into text data.

[0289] "Text data" is digital text in a format that can be visually displayed and is converted from audio data.

[0290] "Visually displaying" means presenting text data on a display in a recognizable format to the user.

[0291] "Detecting anomalies" refers to the process of discovering a state that deviates from normal operation within the system and notifying the user of that information.

[0292] "User" refers to an individual who uses the system to acquire or display audio data.

[0293] This invention is a system for acquiring audio data in real time, converting it into text data, and visually displaying it. In particular, it aims to improve the quality of audio data while reducing background noise. The user installs an application on an information terminal such as a smartphone or tablet and utilizes a cloud computing environment. Generally, a microphone built into the information terminal is used to acquire audio data.

[0294] The device reduces background noise from the acquired audio data by applying noise cancellation technology. This process may utilize an audio processor incorporating existing acoustic technologies. The noise-free audio data is amplified in real time and output to the user. A secure protocol is used for transmitting the audio data over the internet.

[0295] The server analyzes the received audio data in the cloud. This analysis utilizes speech recognition technology and natural language processing algorithms, such as a common speech recognition API. The analyzed audio data is returned to the device as text data. On the device, this text data is displayed visually, and the font size and color settings can be adjusted according to the user's preferences.

[0296] Specific examples include students attending lectures and business people participating in meetings. By receiving text information in real time simultaneously with voice input, misunderstandings and missed information can be prevented, enabling efficient information processing. This system is particularly expected to be used by people with hearing impairments and the elderly.

[0297] An example of a prompt for a generative AI model is: "Please explain the process of a speech recognition system that transcribes lectures into text in real time." This allows the present invention to flexibly adapt to various use cases.

[0298] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0299] Step 1:

[0300] The user installs and launches the application on their smartphone or tablet.

[0301] The input is a user action (launching the app).

[0302] As output, the terminal is set to the voice input mode, and a "Voice Input Start" button is displayed on the interface.

[0303] Step 2:

[0304] The terminal receives a tap on the "Voice Input Start" button from the user as an input and starts capturing ambient sound with the built-in microphone.

[0305] As input, there is ambient sound obtained through the built-in microphone.

[0306] Using noise cancellation technology, background noise is reduced from the input voice to generate clean voice data.

[0307] As output, voice data with noise removed is generated and proceeds to the next processing step.

[0308] Step 3:

[0309] The terminal amplifies the noise-removed voice data in real time.

[0310] As input, there is voice data after noise removal.

[0311] ]> As data processing, amplification processing of the signal is performed to enhance the clarity of the voice.

[0312] As output, amplified voice data is generated and output to the user's earphone or speaker in real time.

[0313] Step 4:

[0314] The terminal compresses the amplified voice data and transmits it to a server on the cloud.

[0315] As input, there is amplified voice data.

[0316] As part of the data processing, the audio data is compressed to make it ready for transmission.

[0317] The compressed audio data is sent to the server as output.

[0318] Step 5:

[0319] The server analyzes the audio data it receives.

[0320] The input is audio data sent from the device.

[0321] For data processing, speech recognition technology and natural language processing are used to convert speech data into text data in real time.

[0322] As output, converted character data is generated and sent to the terminal.

[0323] Step 6:

[0324] The terminal visually displays the character data received from the server.

[0325] The input is character data returned from the server.

[0326] Data processing involves adjusting font size, color, and other settings to format the information as optimal visual data for the user.

[0327] As output, formatted text data is displayed on the terminal's screen, allowing the user to verify the information.

[0328] (Application Example 1)

[0329] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0330] It is difficult for elderly and hearing-impaired users to communicate effectively in care settings. There is a need for technology that removes communication barriers by providing not only audio information but also visual information in real time for these users.

[0331] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0332] In this invention, the server includes an acquisition means for acquiring an audio signal, an amplification means for amplifying the audio signal, and a conversion means for converting the amplified audio signal into text in real time. This enables smooth communication between caregivers and elderly individuals by converting audio information into visual information in real time.

[0333] An "audio signal" is an electrical signal that converts sound information transmitted by air vibrations into an electrical signal.

[0334] "Acquisition means" refers to a device or method for collecting audio signals.

[0335] "Amplification means" refers to a device or method for converting an audio signal to a higher volume.

[0336] "Conversion means" refers to a device or method for converting an audio signal into text data.

[0337] "Display means" refers to a device or method for visually displaying the converted text.

[0338] "Care workers" are people who are engaged in jobs that support the lives of elderly people.

[0339] "Elderly people" generally refers to people who are considered to be older.

[0340] "Support measures" refer to devices or methods that facilitate communication between care workers and elderly individuals.

[0341] "Noise" refers to unwanted sounds that are mixed in with the intended audio signal.

[0342] "Notification means" refers to a device or method for informing a user of an anomaly or event.

[0343] The system for carrying out this invention is configured to acquire, amplify, convert to text, and display audio signals in real time, thereby supporting communication between caregivers and elderly individuals. This system mainly includes audio acquisition means, amplification means, conversion means, and display means. The server uses advanced speech recognition technology and natural language processing to accurately convert the acquired audio signals into text data and transmit it to the user's terminal. The terminal displays this as visual information, allowing the user to confirm it both aurally and visually.

[0344] The primary hardware includes the user's smartphone or tablet, as well as earphones and speakers. The software utilizes noise cancellation technology (e.g., Active Noise Cancellation), speech recognition APIs (e.g., Google Cloud Speech-to-Text), and natural language processing libraries (e.g., spaCy). These technologies, combined with the server's advanced processing of audio signals, enable users to receive audio as visual information in real time.

[0345] As a concrete example, consider a scenario in a nursing home where caregivers give instructions regarding meals and medication to elderly residents. The caregiver's voice is amplified and converted into text on the resident's device, and displayed as visual information on the screen, making it easier for the resident to accurately understand the instructions.

[0346] An example of a prompt for a generative AI model might be: "Imagine a scenario in a nursing home where care staff are giving instructions to residents. Explain how a real-time speech conversion app can support smooth communication." This prompt would prompt the generative AI model to describe specific support measures and their effects.

[0347] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0348] Step 1:

[0349] The device uses its built-in microphone to acquire ambient audio signals. The acquired audio signal is then used as input, and background noise is removed using noise-canceling technology. This allows for the output of a clear audio signal.

[0350] Step 2:

[0351] The device amplifies the noise-reduced audio signal and outputs it to the user's earphones or speakers. The audio signal is amplified in real time and provided at a volume that is easier for the user to hear.

[0352] Step 3:

[0353] The terminal sends the amplified audio signal to a server in the cloud. The server receives the audio signal as input and converts it into digital text using advanced speech recognition technology. The converted text data is then generated as output data.

[0354] Step 4:

[0355] The server sends the generated text data back to the terminal. The terminal takes the text data as input and displays it visually on the screen. The font size and color are adjusted so that the user can easily read it.

[0356] Step 5:

[0357] Users can confirm instructions from caregivers through displayed text. This allows for communication using both visual and auditory means.

[0358] Step 6:

[0359] The terminal monitors for system anomalies and displays an error message to the user if an anomaly is detected. This allows the user to obtain information to take the next action.

[0360] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0361] This invention combines a system that acquires user voice signals, amplifies them in real time, and transcribes them into text with an emotion engine. By using the emotion engine, it is possible to identify emotions from user speech and enable appropriate interaction.

[0362] The user installs and launches the application on a device such as a smartphone or tablet. This application has the function of acquiring audio signals through the microphone, and the acquired audio signals are processed in real time using noise cancellation technology. The device then amplifies these audio signals, providing the user with clear audio.

[0363] The audio signal is then sent to a server in the cloud, where it is converted into text using speech recognition and natural language processing technologies. During this process, an emotion engine analyzes the user's emotional state from the audio and extracts emotion data. This data indicates the user's current emotional state, and the analysis results are sent to the device.

[0364] The device dynamically adjusts the format of the text it displays based on acquired emotional data, for example, by changing the color and font size according to the user's emotions. Furthermore, when the device detects user stress or anxiety, it can automatically provide voice guidance for relaxation.

[0365] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can hear clear audio through the headset while reducing ambient noise, and simultaneously receive visual feedback in the form of text. Furthermore, if the emotion engine detects that the user is experiencing stress, the device can suggest playing relaxation music. In this way, the present invention provides a system that reduces the user's mental burden and supports smooth communication.

[0366] The following describes the processing flow.

[0367] Step 1:

[0368] The user launches the application on their device and selects voice input mode. This puts the device into a state where it can begin capturing audio.

[0369] Step 2:

[0370] The device uses its built-in microphone to acquire audio signals in real time. During this process, noise cancellation is enabled to remove ambient noise and obtain clear audio data.

[0371] Step 3:

[0372] The device analyzes the acquired audio signal and amplifies it to an appropriate volume for output. The amplified audio is then delivered to the user through earphones or speakers.

[0373] Step 4:

[0374] The device sends the de-noised audio signal to a server in the cloud, requesting data processing for speech recognition and sentiment analysis.

[0375] Step 5:

[0376] The server converts the received audio data into text using speech recognition technology. Simultaneously, it uses an emotion engine to analyze the user's emotional state from their voice and generates the results.

[0377] Step 6:

[0378] The server sends the analyzed text data and sentiment data back to the terminal. Communication is performed with low latency, prioritizing user convenience.

[0379] Step 7:

[0380] When a device displays received text on the screen, it automatically adjusts the formatting based on sentiment data. This includes changing the text color and font size according to the sentiment.

[0381] Step 8:

[0382] If the device detects negative emotions such as stress or anxiety, it will automatically provide voice guidance or music for relaxation. For example, it can play soothing music to help alleviate mental stress.

[0383] Step 9:

[0384] Users can adjust various settings options provided on their device as needed to enjoy a personalized experience. Other background sounds and responses that respond to specific emotions can also be configured.

[0385] (Example 2)

[0386] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0387] In modern society, users are required to handle audio information in various environments and understand the speaker's emotions within those contexts. However, conventional systems are susceptible to noise, making it difficult to accurately analyze emotional states. Furthermore, they lacked the functionality to provide appropriate feedback in response to the user's emotions. This invention aims to solve these problems and realize emotion-responsive interaction.

[0388] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0389] In this invention, the server includes means for acquiring audio information, means for amplifying the audio information, means for converting the amplified audio information into text information in real time, means for analyzing emotional information, and means for dynamically adjusting the display format based on the emotional information. This makes it possible to accurately analyze the emotional state of the user's speech and provide appropriate feedback while removing noise from the audio signal.

[0390] "Audio information" refers to sound waves represented as digital or analog signals.

[0391] "Means" refers to the equipment, apparatus, or method used to achieve a particular purpose.

[0392] "Amplification" refers to the process of increasing the strength or magnitude of a signal, making it clearer.

[0393] "Real-time" refers to a process that proceeds almost instantly, with virtually no time delay.

[0394] "Textual information" refers to a set of symbols and characters used to visually represent language.

[0395] "Display format" refers to the design and style that indicates how information and data are visually arranged.

[0396] "Dynamic adjustment" refers to the automatic modification or correction of information in response to changes in circumstances or conditions.

[0397] "Analysis" is the process of breaking down complex information or data into an easily understandable form and revealing its meaning and structure.

[0398] This invention is a system that enhances user interaction by acquiring audio information, amplifying it, converting it into text information, and further performing emotion analysis. Specifically, it is implemented as follows.

[0399] The user installs a dedicated application on their mobile device and starts the system by launching this application. This application uses the device's microphone to acquire voice information and utilizes noise cancellation technology to eliminate ambient noise, thus maintaining a clear voice signal.

[0400] Next, the device amplifies the acquired audio signal in real time, providing the user with clearer audio. This process ensures that the audio signal information is not lost, allowing the next step to be completed without any loss of quality.

[0401] The voice signal is transmitted via the internet to a server in the cloud. The server uses speech recognition technology to convert the voice signal into text information. Furthermore, a generative AI model is used to analyze and extract user emotional information from the voice. The information obtained from this emotional analysis is used to reflect and predict the user's psychological state.

[0402] The terminal dynamically adjusts the format of the displayed text information based on emotional information received from the server. For example, if the system determines that the user is calm, it can change the color scheme to a gentle blue, while if the user is excited, it can change to a red color scheme that indicates energy. This allows the system to provide a more user-friendly display.

[0403] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can use a headset to reduce ambient noise, hear clear audio, and simultaneously view visual text information on their device screen. Furthermore, if the user experiences stress, the system can provide relaxation guidance, creating a pleasant experience.

[0404] An example of a prompt statement would be, "I'm feeling stressed during my commute; please suggest some relaxation music." In this way, the present invention provides an effective method for understanding and promoting a user's emotions.

[0405] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0406] Step 1:

[0407] The user launches the application on their mobile device. As input, the user speaks into the device's microphone. The device acquires this audio signal and uses noise-canceling technology to remove unwanted ambient noise. The output is clear audio data.

[0408] Step 2:

[0409] Based on the acquired audio data, the device amplifies the audio signal, providing clear audio to the user in real time. The input is a noise-free audio signal, and the output is amplified audio. This amplification ensures that the user receives all the audio details clearly.

[0410] Step 3:

[0411] The terminal transmits amplified audio data to the server via the internet. The amplified audio signal, as input, becomes the data to be transmitted. The output is confirmation data after transmission is complete. This prepares the server to begin audio analysis.

[0412] Step 4:

[0413] The server uses speech recognition technology with the received audio data to convert speech into text. The input is the transmitted audio data, and the output is text data. The speech recognition engine records the spoken content as text.

[0414] Step 5:

[0415] The server uses a generative AI model to analyze user emotional information from text data. The input is text data, and the output is emotional information. The emotion engine identifies the user's psychological state and extracts the necessary emotional parameters.

[0416] Step 6:

[0417] Based on the emotional information, the server sends the analysis results to the terminal. The input is the analyzed emotional information, and the output is the emotional parameters sent to the terminal. This allows the terminal to prepare to display a response to the user.

[0418] Step 7:

[0419] The terminal dynamically adjusts the format of the text information displayed on the screen using the received emotion information. The input is emotion parameters received from the server, and the output is text information in the adjusted display format. Specifically, it changes the font color and size to display information according to the emotion.

[0420] Step 8:

[0421] Based on emotional information, the device detects user stress and anxiety and provides relaxation-enhancing audio guidance. Input is emotional information and the detection result of the user's stress level; output is playback of relaxation audio. This reduces the user's mental burden.

[0422] (Application Example 2)

[0423] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0424] A challenge exists in that when elderly people communicate with others, they may not receive appropriate responses that address their feelings, potentially increasing their stress and anxiety. While efficient and emotionally responsive support is needed in care settings, current technology has made it difficult to fully achieve this.

[0425] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0426] In this invention, the server includes a device for acquiring audio signals, a processing device for analyzing emotions, and a device for providing visual displays corresponding to the user's emotions. This makes it possible to analyze the emotions of elderly people and provide visual displays and audio guidance corresponding to those emotions.

[0427] A "speech signal" is a representation of sound as an electrical signal, and typically includes the content of human speech.

[0428] An "device" is a machine or system designed to achieve a specific function or purpose.

[0429] "Amplification" is the process of increasing the signal strength, and it is a process performed to make audio signals clearer.

[0430] "Real-time" refers to a state where processing is performed instantly without delay.

[0431] A "text" is a linguistic expression that uses words arranged according to certain rules to convey a specific meaning.

[0432] "Emotion" is a psychological state of a human being, and includes feelings such as joy and sadness.

[0433] "Analysis" is the process of breaking down an object into its constituent parts and understanding its properties.

[0434] A "processing device" is a device that manipulates or converts given data or signals according to instructions.

[0435] "Visual representation" means showing information in a way that is visible to the eye.

[0436] This invention is a system that allows users to receive real-time emotion analysis via voice signals and receive visual feedback and relaxation guidance. First, the terminal acquires the user's voice signal through a microphone. This terminal incorporates noise-canceling technology, which removes external noise and clarifies the voice signal. The voice signal is amplified and then transmitted to a cloud server.

[0437] When the server receives an audio signal, it uses speech recognition software to convert the signal into text. Next, the server uses an emotion analysis engine to identify the user's emotional state based on this text. This analysis may involve third-party speech recognition platforms or natural language processing technologies (e.g., OpenAI's GPT model).

[0438] The results of the emotion analysis are returned to the user's device, which then displays them visually. Furthermore, if stress or anxiety is detected, the device provides relaxation-enhancing audio guidance or music. This allows for appropriate responses tailored to the user's emotional state.

[0439] As a concrete example, consider a situation where an elderly person is conversing with a caregiver in a care facility. Using this system, the elderly person's voice is instantly converted into text, and their emotional state is analyzed. For example, if the emotional analysis determines that the elderly person is feeling anxious, the device can display a visually reassuring message and play relaxing music. This system allows caregivers to respond immediately to the elderly person's mental state.

[0440] An example of a prompt is, "Using this system, how would you like to support the emotional changes of elderly individuals during their daily communication?" Based on this prompt, the generative AI model can construct appropriate interactions that respond to their emotions.

[0441] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0442] Step 1:

[0443] The device acquires audio signals using a microphone. The input is the user's speech, and the output is digitized audio data. This data is processed into a clear audio signal by removing noise using noise cancellation technology.

[0444] Step 2:

[0445] The terminal sends a clear audio signal to the server. The input is a noise-free audio signal, and the output is the signal transfer to the cloud server. The server receives this signal and prepares for the next processing step.

[0446] Step 3:

[0447] The server uses a speech recognition engine to convert the audio signal into text. The input is the transmitted audio signal, and the output is the text data corresponding to that audio. This text is generated using technologies such as generative AI models.

[0448] Step 4:

[0449] The server uses an emotion analysis engine to identify the user's emotional state based on the converted text. The input is text data, and the output is emotion data. The server utilizes natural language processing techniques to analyze emotions.

[0450] Step 5:

[0451] The server instructs the system to generate visual feedback corresponding to the user's emotional state, and, if necessary, relaxation sounds. The input is emotional data, and the output is the instructed feedback content. Prompts may also be used in this process.

[0452] Step 6:

[0453] The terminal provides the user with visual displays and audio guidance based on feedback instructions from the server. The input is the instructions from the server, and the output is the user-viewable screen display and the audible relaxation audio.

[0454] Step 7:

[0455] The user follows the provided feedback, confirms the instructed actions, and performs relaxation as needed. Input is visual and auditory feedback, and output is the user's response or change in internal feelings.

[0456] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0457] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0458] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0459] [Third Embodiment]

[0460] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0461] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0462] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0463] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0464] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0465] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0466] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0467] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0468] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0469] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0470] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0471] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0472] This invention provides a system for acquiring audio signals, amplifying them in real time, and transcribing them into text. Here, the program's processing for effectively handling audio signals is described in natural language.

[0473] First, the user installs and launches the application on a device such as a smartphone or tablet. To begin voice input, the device uses its built-in microphone to capture ambient sounds. At this point, noise cancellation technology is used to reduce background noise and process the sound to obtain clear audio.

[0474] Next, the audio signal acquired by the device is amplified in real time and output to the user's earphones or speakers to improve sound quality. The noise-reduced audio signal is then sent to a server in the cloud for speech recognition.

[0475] The server uses advanced speech recognition technology and natural language processing to convert received audio signals into text data. This converted text data is sent back to the terminal in real time, where the terminal visually displays the text. Here, the font size and color are adjusted to make it easy for elderly people to read. By providing information through both sight and sound, the quality of communication is improved.

[0476] As a concrete example, consider a student attending a lecture. This student can listen to the lecturer's explanation in audio format and simultaneously visualize the content as text. In this case, the text is updated in real time and displayed in sync with the audio, allowing the student to easily understand the lecture content and take notes.

[0477] If the terminal detects an abnormality or error, it can display an error message to the user and prompt them to re-enter the information, thus maintaining smooth communication at all times. In this way, the invention not only improves the quality of life for elderly people with hearing loss and those with hearing impairments, but is also expected to have applications in various environments.

[0478] The following describes the processing flow.

[0479] Step 1:

[0480] The user launches an application on their device and operates the interface to initiate voice input. This starts the collection of the voice signal.

[0481] Step 2:

[0482] The device uses its built-in microphone to acquire audio signals in real time. Noise cancellation technology is then applied to process the signal to obtain an acoustically clear signal.

[0483] Step 3:

[0484] The device amplifies the audio signal it receives using a specified algorithm, adjusts the volume, and then outputs it to the user's earphones or speaker.

[0485] Step 4:

[0486] The device sends the noise-reduced audio signal to a server in the cloud for speech recognition processing. The transmission is performed using a secure communication protocol.

[0487] Step 5:

[0488] The server converts the received audio signal into text data using speech recognition technology. If necessary, natural language processing is applied to analyze the sentence structure and generate accurate text.

[0489] Step 6:

[0490] The server sends the converted text back to the terminal in real time. Optimization is performed here to minimize communication delays.

[0491] Step 7:

[0492] The device displays received text on the screen. The display format is customizable for user readability, supporting adjustments to font size, background color, and other settings.

[0493] Step 8:

[0494] Users can adjust volume and text display settings within the application as needed, optimizing them for their individual needs.

[0495] Step 9:

[0496] If the device detects a system anomaly, it immediately notifies the user and guides them to try voice input again if necessary. Seamless error handling during operation maintains system stability.

[0497] (Example 1)

[0498] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0499] There is a need to convert audio data into text data in real time and to quickly notify users when an anomaly is detected. In particular, the challenge is to accurately process information while effectively reducing background noise and improving sound quality, thereby enhancing the quality of life.

[0500] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0501] In this invention, the server includes means for transmitting audio data, means for analyzing the audio data on the cloud and converting it into text data, and means for receiving the text data and displaying it visually. This combines advanced background noise reduction technology with a process for improving sound quality, enabling accurate real-time information provision and rapid notification in the event of anomalies.

[0502] "Audio data" refers to data extracted from audio signals and represented in digital format.

[0503] "Device" refers to a combination of hardware and software for acquiring, processing, or displaying audio data.

[0504] "Background noise reduction" refers to the process of removing unwanted noise from audio data and emphasizing the main audio portion.

[0505] "Amplifying" is the process of increasing the signal strength of audio data and improving its clarity.

[0506] "Transmission" refers to the act of transferring processed audio data to another device or cloud server.

[0507] "Analysis on the cloud" refers to analyzing audio data in a cloud computing environment and converting it into text data.

[0508] "Text data" refers to digital text converted from audio data and displayed in a visually recognizable format.

[0509] "Visual display" means presenting text data on a display in a format that is recognizable to the user.

[0510] "Detecting anomalies" refers to the process of discovering a state within a system that deviates from normal operation and notifying the user of that information.

[0511] "User" refers to an individual who uses the system to acquire or display voice data.

[0512] This invention is a system for acquiring audio data in real time, converting it into text data, and displaying it visually. Its primary purpose is to improve the quality of the audio data while reducing background noise. Users install an application on information terminals such as smartphones and tablets and utilize a cloud computing environment. Typically, the microphone built into the information terminal is used to acquire the audio data.

[0513] The device reduces background noise from the acquired audio data by applying noise cancellation technology. This process may utilize an audio processor incorporating existing acoustic technologies. The noise-free audio data is amplified in real time and output to the user. A secure protocol is used for transmitting the audio data over the internet.

[0514] The server analyzes the received audio data in the cloud. This analysis utilizes speech recognition technology and natural language processing algorithms, such as a common speech recognition API. The analyzed audio data is returned to the device as text data. On the device, this text data is displayed visually, and the font size and color settings can be adjusted according to the user's preferences.

[0515] Specific examples include students attending lectures and business people participating in meetings. By receiving text information in real time simultaneously with voice input, misunderstandings and missed information can be prevented, enabling efficient information processing. This system is particularly expected to be used by people with hearing impairments and the elderly.

[0516] An example of a prompt for a generative AI model is: "Please explain the process of a speech recognition system that transcribes lectures into text in real time." This allows the present invention to flexibly adapt to various use cases.

[0517] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0518] Step 1:

[0519] The user installs and launches the application on their smartphone or tablet.

[0520] The input is a user action (launching the app).

[0521] As an output, the device is set to voice input mode, and a "Start Voice Input" button is displayed on the interface.

[0522] Step 2:

[0523] The device receives a tap of the "Start Voice Input" button as input from the user and begins capturing ambient sounds with its built-in microphone.

[0524] The input includes ambient sounds acquired through the built-in microphone.

[0525] Using noise cancellation technology, background noise is reduced from the input audio, generating clean audio data.

[0526] The output is audio data with noise removed, and the process proceeds to the next processing step.

[0527] Step 3:

[0528] The device amplifies the noise-reduced audio data in real time.

[0529] The input is audio data after noise reduction.

[0530] As part of the data processing, signal amplification is performed to improve the clarity of the audio.

[0531] The output is amplified audio data, which is then output in real time to the user's earphones or speakers.

[0532] Step 4:

[0533] The device compresses the amplified audio data and sends it to a server in the cloud.

[0534] The input is amplified audio data.

[0535] As part of the data processing, the audio data is compressed to make it ready for transmission.

[0536] The compressed audio data is sent to the server as output.

[0537] Step 5:

[0538] The server analyzes the audio data it receives.

[0539] The input is audio data sent from the device.

[0540] For data processing, speech recognition technology and natural language processing are used to convert speech data into text data in real time.

[0541] As output, converted character data is generated and sent to the terminal.

[0542] Step 6:

[0543] The terminal visually displays the character data received from the server.

[0544] The input is character data returned from the server.

[0545] Data processing involves adjusting font size, color, and other settings to format the information as optimal visual data for the user.

[0546] As output, formatted text data is displayed on the terminal's screen, allowing the user to verify the information.

[0547] (Application Example 1)

[0548] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0549] It is difficult for elderly and hearing-impaired users to communicate effectively in care settings. There is a need for technology that removes communication barriers by providing not only audio information but also visual information in real time for these users.

[0550] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0551] In this invention, the server includes an acquisition means for acquiring an audio signal, an amplification means for amplifying the audio signal, and a conversion means for converting the amplified audio signal into text in real time. This enables smooth communication between caregivers and elderly individuals by converting audio information into visual information in real time.

[0552] An "audio signal" is an electrical signal that converts sound information transmitted by air vibrations into an electrical signal.

[0553] "Acquisition means" refers to a device or method for collecting audio signals.

[0554] "Amplification means" refers to a device or method for converting an audio signal to a higher volume.

[0555] "Conversion means" refers to a device or method for converting an audio signal into text data.

[0556] "Display means" refers to a device or method for visually displaying the converted text.

[0557] "Care workers" are people who are engaged in jobs that support the lives of elderly people.

[0558] "Elderly people" generally refers to people who are considered to be older.

[0559] "Support measures" refer to devices or methods that facilitate communication between care workers and elderly individuals.

[0560] "Noise" refers to unwanted sounds that are mixed in with the intended audio signal.

[0561] "Notification means" refers to a device or method for informing a user of an anomaly or event.

[0562] The system for carrying out this invention is configured to acquire, amplify, convert to text, and display audio signals in real time, thereby supporting communication between caregivers and elderly individuals. This system mainly includes audio acquisition means, amplification means, conversion means, and display means. The server uses advanced speech recognition technology and natural language processing to accurately convert the acquired audio signals into text data and transmit it to the user's terminal. The terminal displays this as visual information, allowing the user to confirm it both aurally and visually.

[0563] The primary hardware includes the user's smartphone or tablet, as well as earphones and speakers. The software utilizes noise cancellation technology (e.g., Active Noise Cancellation), speech recognition APIs (e.g., Google Cloud Speech-to-Text), and natural language processing libraries (e.g., spaCy). These technologies, combined with the server's advanced processing of audio signals, enable users to receive audio as visual information in real time.

[0564] As a concrete example, consider a scenario in a nursing home where caregivers give instructions regarding meals and medication to elderly residents. The caregiver's voice is amplified and converted into text on the resident's device, and displayed as visual information on the screen, making it easier for the resident to accurately understand the instructions.

[0565] An example of a prompt for a generative AI model might be: "Imagine a scenario in a nursing home where care staff are giving instructions to residents. Explain how a real-time speech conversion app can support smooth communication." This prompt would prompt the generative AI model to describe specific support measures and their effects.

[0566] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0567] Step 1:

[0568] The device uses its built-in microphone to acquire ambient audio signals. The acquired audio signal is then used as input, and background noise is removed using noise-canceling technology. This allows for the output of a clear audio signal.

[0569] Step 2:

[0570] The device amplifies the noise-reduced audio signal and outputs it to the user's earphones or speakers. The audio signal is amplified in real time and provided at a volume that is easier for the user to hear.

[0571] Step 3:

[0572] The terminal sends the amplified audio signal to a server in the cloud. The server receives the audio signal as input and converts it into digital text using advanced speech recognition technology. The converted text data is then generated as output data.

[0573] Step 4:

[0574] The server sends the generated text data back to the terminal. The terminal takes the text data as input and displays it visually on the screen. The font size and color are adjusted so that the user can easily read it.

[0575] Step 5:

[0576] Users can confirm instructions from caregivers through displayed text. This allows for communication using both visual and auditory means.

[0577] Step 6:

[0578] The terminal monitors for system anomalies and displays an error message to the user if an anomaly is detected. This allows the user to obtain information to take the next action.

[0579] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0580] This invention combines a system that acquires user voice signals, amplifies them in real time, and transcribes them into text with an emotion engine. By using the emotion engine, it is possible to identify emotions from user speech and enable appropriate interaction.

[0581] The user installs and launches the application on a device such as a smartphone or tablet. This application has the function of acquiring audio signals through the microphone, and the acquired audio signals are processed in real time using noise cancellation technology. The device then amplifies these audio signals, providing the user with clear audio.

[0582] The audio signal is then sent to a server in the cloud, where it is converted into text using speech recognition and natural language processing technologies. During this process, an emotion engine analyzes the user's emotional state from the audio and extracts emotion data. This data indicates the user's current emotional state, and the analysis results are sent to the device.

[0583] The device dynamically adjusts the format of the text it displays based on acquired emotional data, for example, by changing the color and font size according to the user's emotions. Furthermore, when the device detects user stress or anxiety, it can automatically provide voice guidance for relaxation.

[0584] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can hear clear audio through the headset while reducing ambient noise, and simultaneously receive visual feedback in the form of text. Furthermore, if the emotion engine detects that the user is experiencing stress, the device can suggest playing relaxation music. In this way, the present invention provides a system that reduces the user's mental burden and supports smooth communication.

[0585] The following describes the processing flow.

[0586] Step 1:

[0587] The user launches the application on their device and selects voice input mode. This puts the device into a state where it can begin capturing audio.

[0588] Step 2:

[0589] The device uses its built-in microphone to acquire audio signals in real time. During this process, noise cancellation is enabled to remove ambient noise and obtain clear audio data.

[0590] Step 3:

[0591] The device analyzes the acquired audio signal and amplifies it to an appropriate volume for output. The amplified audio is then delivered to the user through earphones or speakers.

[0592] Step 4:

[0593] The device sends the de-noised audio signal to a server in the cloud, requesting data processing for speech recognition and sentiment analysis.

[0594] Step 5:

[0595] The server converts the received audio data into text using speech recognition technology. Simultaneously, it uses an emotion engine to analyze the user's emotional state from their voice and generates the results.

[0596] Step 6:

[0597] The server sends the analyzed text data and sentiment data back to the terminal. Communication is performed with low latency, prioritizing user convenience.

[0598] Step 7:

[0599] When a device displays received text on the screen, it automatically adjusts the formatting based on sentiment data. This includes changing the text color and font size according to the sentiment.

[0600] Step 8:

[0601] If the device detects negative emotions such as stress or anxiety, it will automatically provide voice guidance or music for relaxation. For example, it can play soothing music to help alleviate mental stress.

[0602] Step 9:

[0603] Users can adjust various settings options provided on their device as needed to enjoy a personalized experience. Other background sounds and responses that respond to specific emotions can also be configured.

[0604] (Example 2)

[0605] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0606] In modern society, users are required to handle audio information in various environments and understand the speaker's emotions within those contexts. However, conventional systems are susceptible to noise, making it difficult to accurately analyze emotional states. Furthermore, they lacked the functionality to provide appropriate feedback in response to the user's emotions. This invention aims to solve these problems and realize emotion-responsive interaction.

[0607] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0608] In this invention, the server includes means for acquiring audio information, means for amplifying the audio information, means for converting the amplified audio information into text information in real time, means for analyzing emotional information, and means for dynamically adjusting the display format based on the emotional information. This makes it possible to accurately analyze the emotional state of the user's speech and provide appropriate feedback while removing noise from the audio signal.

[0609] "Audio information" refers to sound waves represented as digital or analog signals.

[0610] "Means" refers to the equipment, apparatus, or method used to achieve a particular purpose.

[0611] "Amplification" refers to the process of increasing the strength or magnitude of a signal, making it clearer.

[0612] "Real-time" refers to a process that proceeds almost instantly, with virtually no time delay.

[0613] "Textual information" refers to a set of symbols and characters used to visually represent language.

[0614] "Display format" refers to the design and style that indicates how information and data are visually arranged.

[0615] "Dynamic adjustment" refers to the automatic modification or correction of information in response to changes in circumstances or conditions.

[0616] "Analysis" is the process of breaking down complex information or data into an easily understandable form and revealing its meaning and structure.

[0617] This invention is a system that enhances user interaction by acquiring audio information, amplifying it, converting it into text information, and further performing emotion analysis. Specifically, it is implemented as follows.

[0618] The user installs a dedicated application on their mobile device and starts the system by launching this application. This application uses the device's microphone to acquire voice information and utilizes noise cancellation technology to eliminate ambient noise, thus maintaining a clear voice signal.

[0619] Next, the device amplifies the acquired audio signal in real time, providing the user with clearer audio. This process ensures that the audio signal information is not lost, allowing the next step to be completed without any loss of quality.

[0620] The voice signal is transmitted via the internet to a server in the cloud. The server uses speech recognition technology to convert the voice signal into text information. Furthermore, a generative AI model is used to analyze and extract user emotional information from the voice. The information obtained from this emotional analysis is used to reflect and predict the user's psychological state.

[0621] The terminal dynamically adjusts the format of the displayed text information based on emotional information received from the server. For example, if the system determines that the user is calm, it can change the color scheme to a gentle blue, while if the user is excited, it can change to a red color scheme that indicates energy. This allows the system to provide a more user-friendly display.

[0622] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can use a headset to reduce ambient noise, hear clear audio, and simultaneously view visual text information on their device screen. Furthermore, if the user experiences stress, the system can provide relaxation guidance, creating a pleasant experience.

[0623] An example of a prompt statement would be, "I'm feeling stressed during my commute; please suggest some relaxation music." In this way, the present invention provides an effective method for understanding and promoting a user's emotions.

[0624] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0625] Step 1:

[0626] The user launches the application on their mobile device. As input, the user speaks into the device's microphone. The device acquires this audio signal and uses noise-canceling technology to remove unwanted ambient noise. The output is clear audio data.

[0627] Step 2:

[0628] Based on the acquired audio data, the device amplifies the audio signal, providing clear audio to the user in real time. The input is a noise-free audio signal, and the output is amplified audio. This amplification ensures that the user receives all the audio details clearly.

[0629] Step 3:

[0630] The terminal transmits amplified audio data to the server via the internet. The amplified audio signal, as input, becomes the data to be transmitted. The output is confirmation data after transmission is complete. This prepares the server to begin audio analysis.

[0631] Step 4:

[0632] The server uses speech recognition technology with the received audio data to convert speech into text. The input is the transmitted audio data, and the output is text data. The speech recognition engine records the spoken content as text.

[0633] Step 5:

[0634] The server uses a generative AI model to analyze user emotional information from text data. The input is text data, and the output is emotional information. The emotion engine identifies the user's psychological state and extracts the necessary emotional parameters.

[0635] Step 6:

[0636] Based on the emotional information, the server sends the analysis results to the terminal. The input is the analyzed emotional information, and the output is the emotional parameters sent to the terminal. This allows the terminal to prepare to display a response to the user.

[0637] Step 7:

[0638] The terminal dynamically adjusts the format of the text information displayed on the screen using the received emotion information. The input is emotion parameters received from the server, and the output is text information in the adjusted display format. Specifically, it changes the font color and size to display information according to the emotion.

[0639] Step 8:

[0640] Based on emotional information, the device detects user stress and anxiety and provides relaxation-enhancing audio guidance. Input is emotional information and the detection result of the user's stress level; output is playback of relaxation audio. This reduces the user's mental burden.

[0641] (Application Example 2)

[0642] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0643] A challenge exists in that when elderly people communicate with others, they may not receive appropriate responses that address their feelings, potentially increasing their stress and anxiety. While efficient and emotionally responsive support is needed in care settings, current technology has made it difficult to fully achieve this.

[0644] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0645] In this invention, the server includes a device for acquiring audio signals, a processing device for analyzing emotions, and a device for providing visual displays corresponding to the user's emotions. This makes it possible to analyze the emotions of elderly people and provide visual displays and audio guidance corresponding to those emotions.

[0646] A "speech signal" is a representation of sound as an electrical signal, and typically includes the content of human speech.

[0647] An "device" is a machine or system designed to achieve a specific function or purpose.

[0648] "Amplification" is the process of increasing the signal strength, and it is a process performed to make audio signals clearer.

[0649] "Real-time" refers to a state where processing is performed instantly without delay.

[0650] A "text" is a linguistic expression that uses words arranged according to certain rules to convey a specific meaning.

[0651] "Emotion" is a psychological state of a human being, and includes feelings such as joy and sadness.

[0652] "Analysis" is the process of breaking down an object into its constituent parts and understanding its properties.

[0653] A "processing device" is a device that manipulates or converts given data or signals according to instructions.

[0654] "Visual representation" means showing information in a way that is visible to the eye.

[0655] This invention is a system that allows users to receive real-time emotion analysis via voice signals and receive visual feedback and relaxation guidance. First, the terminal acquires the user's voice signal through a microphone. This terminal incorporates noise-canceling technology, which removes external noise and clarifies the voice signal. The voice signal is amplified and then transmitted to a cloud server.

[0656] When the server receives an audio signal, it uses speech recognition software to convert the signal into text. Next, the server uses an emotion analysis engine to identify the user's emotional state based on this text. This analysis may involve third-party speech recognition platforms or natural language processing technologies (e.g., OpenAI's GPT model).

[0657] The results of the emotion analysis are returned to the user's device, which then displays them visually. Furthermore, if stress or anxiety is detected, the device provides relaxation-enhancing audio guidance or music. This allows for appropriate responses tailored to the user's emotional state.

[0658] As a concrete example, consider a situation where an elderly person is conversing with a caregiver in a care facility. Using this system, the elderly person's voice is instantly converted into text, and their emotional state is analyzed. For example, if the emotional analysis determines that the elderly person is feeling anxious, the device can display a visually reassuring message and play relaxing music. This system allows caregivers to respond immediately to the elderly person's mental state.

[0659] An example of a prompt is, "Using this system, how would you like to support the emotional changes of elderly individuals during their daily communication?" Based on this prompt, the generative AI model can construct appropriate interactions that respond to their emotions.

[0660] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0661] Step 1:

[0662] The device acquires audio signals using a microphone. The input is the user's speech, and the output is digitized audio data. This data is processed into a clear audio signal by removing noise using noise cancellation technology.

[0663] Step 2:

[0664] The terminal sends a clear audio signal to the server. The input is a noise-free audio signal, and the output is the signal transfer to the cloud server. The server receives this signal and prepares for the next processing step.

[0665] Step 3:

[0666] The server uses a speech recognition engine to convert the audio signal into text. The input is the transmitted audio signal, and the output is the text data corresponding to that audio. This text is generated using technologies such as generative AI models.

[0667] Step 4:

[0668] The server uses an emotion analysis engine to identify the user's emotional state based on the converted text. The input is text data, and the output is emotion data. The server utilizes natural language processing techniques to analyze emotions.

[0669] Step 5:

[0670] The server instructs the system to generate visual feedback corresponding to the user's emotional state, and, if necessary, relaxation sounds. The input is emotional data, and the output is the instructed feedback content. Prompts may also be used in this process.

[0671] Step 6:

[0672] The terminal provides the user with visual displays and audio guidance based on feedback instructions from the server. The input is the instructions from the server, and the output is the user-viewable screen display and the audible relaxation audio.

[0673] Step 7:

[0674] The user follows the provided feedback, confirms the instructed actions, and performs relaxation as needed. Input is visual and auditory feedback, and output is the user's response or change in internal feelings.

[0675] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0676] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0677] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0678] [Fourth Embodiment]

[0679] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0680] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0681] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0682] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0683] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0684] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0685] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0686] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0687] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0688] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0689] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0690] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0691] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0692] This invention provides a system for acquiring audio signals, amplifying them in real time, and transcribing them into text. Here, the program's processing for effectively handling audio signals is described in natural language.

[0693] First, the user installs and launches the application on a device such as a smartphone or tablet. To begin voice input, the device uses its built-in microphone to capture ambient sounds. At this point, noise cancellation technology is used to reduce background noise and process the sound to obtain clear audio.

[0694] Next, the audio signal acquired by the device is amplified in real time and output to the user's earphones or speakers to improve sound quality. The noise-reduced audio signal is then sent to a server in the cloud for speech recognition.

[0695] The server uses advanced speech recognition technology and natural language processing to convert received audio signals into text data. This converted text data is sent back to the terminal in real time, where the terminal visually displays the text. Here, the font size and color are adjusted to make it easy for elderly people to read. By providing information through both sight and sound, the quality of communication is improved.

[0696] As a concrete example, consider a student attending a lecture. This student can listen to the lecturer's explanation in audio format and simultaneously visualize the content as text. In this case, the text is updated in real time and displayed in sync with the audio, allowing the student to easily understand the lecture content and take notes.

[0697] If the terminal detects an abnormality or error, it can display an error message to the user and prompt them to re-enter the information, thus maintaining smooth communication at all times. In this way, the invention not only improves the quality of life for elderly people with hearing loss and those with hearing impairments, but is also expected to have applications in various environments.

[0698] The following describes the processing flow.

[0699] Step 1:

[0700] The user launches an application on their device and operates the interface to initiate voice input. This starts the collection of the voice signal.

[0701] Step 2:

[0702] The device uses its built-in microphone to acquire audio signals in real time. Noise cancellation technology is then applied to process the signal to obtain an acoustically clear signal.

[0703] Step 3:

[0704] The device amplifies the audio signal it receives using a specified algorithm, adjusts the volume, and then outputs it to the user's earphones or speaker.

[0705] Step 4:

[0706] The device sends the noise-reduced audio signal to a server in the cloud for speech recognition processing. The transmission is performed using a secure communication protocol.

[0707] Step 5:

[0708] The server converts the received audio signal into text data using speech recognition technology. If necessary, natural language processing is applied to analyze the sentence structure and generate accurate text.

[0709] Step 6:

[0710] The server sends the converted text back to the terminal in real time. Optimization is performed here to minimize communication delays.

[0711] Step 7:

[0712] The device displays received text on the screen. The display format is customizable for user readability, supporting adjustments to font size, background color, and other settings.

[0713] Step 8:

[0714] Users can adjust volume and text display settings within the application as needed, optimizing them for their individual needs.

[0715] Step 9:

[0716] If the device detects a system anomaly, it immediately notifies the user and guides them to try voice input again if necessary. Seamless error handling during operation maintains system stability.

[0717] (Example 1)

[0718] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0719] There is a need to convert audio data into text data in real time and to quickly notify users when an anomaly is detected. In particular, the challenge is to accurately process information while effectively reducing background noise and improving sound quality, thereby enhancing the quality of life.

[0720] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0721] In this invention, the server includes means for transmitting audio data, means for analyzing the audio data on the cloud and converting it into text data, and means for receiving the text data and displaying it visually. This combines advanced background noise reduction technology with a process for improving sound quality, enabling accurate real-time information provision and rapid notification in the event of anomalies.

[0722] "Audio data" refers to data extracted from audio signals and represented in digital format.

[0723] "Device" refers to a combination of hardware and software for acquiring, processing, or displaying audio data.

[0724] "Background noise reduction" refers to the process of removing unwanted noise from audio data and emphasizing the main audio portion.

[0725] "Amplifying" is the process of increasing the signal strength of audio data and improving its clarity.

[0726] "Transmission" refers to the act of transferring processed audio data to another device or cloud server.

[0727] "Analysis on the cloud" refers to analyzing audio data in a cloud computing environment and converting it into text data.

[0728] "Text data" refers to digital text converted from audio data and displayed in a visually recognizable format.

[0729] "Visual display" means presenting text data on a display in a format that is recognizable to the user.

[0730] "Detecting anomalies" refers to the process of discovering a state within a system that deviates from normal operation and notifying the user of that information.

[0731] "User" refers to an individual who uses the system to acquire or display voice data.

[0732] This invention is a system for acquiring audio data in real time, converting it into text data, and displaying it visually. Its primary purpose is to improve the quality of the audio data while reducing background noise. Users install an application on information terminals such as smartphones and tablets and utilize a cloud computing environment. Typically, the microphone built into the information terminal is used to acquire the audio data.

[0733] The device reduces background noise from the acquired audio data by applying noise cancellation technology. This process may utilize an audio processor incorporating existing acoustic technologies. The noise-free audio data is amplified in real time and output to the user. A secure protocol is used for transmitting the audio data over the internet.

[0734] The server analyzes the received audio data in the cloud. This analysis utilizes speech recognition technology and natural language processing algorithms, such as a common speech recognition API. The analyzed audio data is returned to the device as text data. On the device, this text data is displayed visually, and the font size and color settings can be adjusted according to the user's preferences.

[0735] Specific examples include students attending lectures and business people participating in meetings. By receiving text information in real time simultaneously with voice input, misunderstandings and missed information can be prevented, enabling efficient information processing. This system is particularly expected to be used by people with hearing impairments and the elderly.

[0736] An example of a prompt for a generative AI model is: "Please explain the process of a speech recognition system that transcribes lectures into text in real time." This allows the present invention to flexibly adapt to various use cases.

[0737] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0738] Step 1:

[0739] The user installs and launches the application on their smartphone or tablet.

[0740] The input is a user action (launching the app).

[0741] As an output, the device is set to voice input mode, and a "Start Voice Input" button is displayed on the interface.

[0742] Step 2:

[0743] The device receives a tap of the "Start Voice Input" button as input from the user and begins capturing ambient sounds with its built-in microphone.

[0744] The input includes ambient sounds acquired through the built-in microphone.

[0745] Using noise cancellation technology, background noise is reduced from the input audio, generating clean audio data.

[0746] The output is audio data with noise removed, and the process proceeds to the next processing step.

[0747] Step 3:

[0748] The device amplifies the noise-reduced audio data in real time.

[0749] The input is audio data after noise reduction.

[0750] As part of the data processing, signal amplification is performed to improve the clarity of the audio.

[0751] The output is amplified audio data, which is then output in real time to the user's earphones or speakers.

[0752] Step 4:

[0753] The device compresses the amplified audio data and sends it to a server in the cloud.

[0754] The input is amplified audio data.

[0755] As part of the data processing, the audio data is compressed to make it ready for transmission.

[0756] The compressed audio data is sent to the server as output.

[0757] Step 5:

[0758] The server analyzes the audio data it receives.

[0759] The input is audio data sent from the device.

[0760] For data processing, speech recognition technology and natural language processing are used to convert speech data into text data in real time.

[0761] As output, converted character data is generated and sent to the terminal.

[0762] Step 6:

[0763] The terminal visually displays the character data received from the server.

[0764] The input is character data returned from the server.

[0765] Data processing involves adjusting font size, color, and other settings to format the information as optimal visual data for the user.

[0766] As output, formatted text data is displayed on the terminal's screen, allowing the user to verify the information.

[0767] (Application Example 1)

[0768] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0769] It is difficult for elderly and hearing-impaired users to communicate effectively in care settings. There is a need for technology that removes communication barriers by providing not only audio information but also visual information in real time for these users.

[0770] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0771] In this invention, the server includes an acquisition means for acquiring an audio signal, an amplification means for amplifying the audio signal, and a conversion means for converting the amplified audio signal into text in real time. This enables smooth communication between caregivers and elderly individuals by converting audio information into visual information in real time.

[0772] An "audio signal" is an electrical signal that converts sound information transmitted by air vibrations into an electrical signal.

[0773] "Acquisition means" refers to a device or method for collecting audio signals.

[0774] "Amplification means" refers to a device or method for converting an audio signal to a higher volume.

[0775] "Conversion means" refers to a device or method for converting an audio signal into text data.

[0776] "Display means" refers to a device or method for visually displaying the converted text.

[0777] "Care workers" are people who are engaged in jobs that support the lives of elderly people.

[0778] "Elderly people" generally refers to people who are considered to be older.

[0779] "Support measures" refer to devices or methods that facilitate communication between care workers and elderly individuals.

[0780] "Noise" refers to unwanted sounds that are mixed in with the intended audio signal.

[0781] "Notification means" refers to a device or method for informing a user of an anomaly or event.

[0782] The system for carrying out this invention is configured to acquire, amplify, convert to text, and display audio signals in real time, thereby supporting communication between caregivers and elderly individuals. This system mainly includes audio acquisition means, amplification means, conversion means, and display means. The server uses advanced speech recognition technology and natural language processing to accurately convert the acquired audio signals into text data and transmit it to the user's terminal. The terminal displays this as visual information, allowing the user to confirm it both aurally and visually.

[0783] The primary hardware includes the user's smartphone or tablet, as well as earphones and speakers. The software utilizes noise cancellation technology (e.g., Active Noise Cancellation), speech recognition APIs (e.g., Google Cloud Speech-to-Text), and natural language processing libraries (e.g., spaCy). These technologies, combined with the server's advanced processing of audio signals, enable users to receive audio as visual information in real time.

[0784] As a concrete example, consider a scenario in a nursing home where caregivers give instructions regarding meals and medication to elderly residents. The caregiver's voice is amplified and converted into text on the resident's device, and displayed as visual information on the screen, making it easier for the resident to accurately understand the instructions.

[0785] An example of a prompt for a generative AI model might be: "Imagine a scenario in a nursing home where care staff are giving instructions to residents. Explain how a real-time speech conversion app can support smooth communication." This prompt would prompt the generative AI model to describe specific support measures and their effects.

[0786] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0787] Step 1:

[0788] The device uses its built-in microphone to acquire ambient audio signals. The acquired audio signal is then used as input, and background noise is removed using noise-canceling technology. This allows for the output of a clear audio signal.

[0789] Step 2:

[0790] The device amplifies the noise-reduced audio signal and outputs it to the user's earphones or speakers. The audio signal is amplified in real time and provided at a volume that is easier for the user to hear.

[0791] Step 3:

[0792] The terminal sends the amplified audio signal to a server in the cloud. The server receives the audio signal as input and converts it into digital text using advanced speech recognition technology. The converted text data is then generated as output data.

[0793] Step 4:

[0794] The server sends the generated text data back to the terminal. The terminal takes the text data as input and displays it visually on the screen. The font size and color are adjusted so that the user can easily read it.

[0795] Step 5:

[0796] Users can confirm instructions from caregivers through displayed text. This allows for communication using both visual and auditory means.

[0797] Step 6:

[0798] The terminal monitors for system anomalies and displays an error message to the user if an anomaly is detected. This allows the user to obtain information to take the next action.

[0799] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0800] This invention combines a system that acquires user voice signals, amplifies them in real time, and transcribes them into text with an emotion engine. By using the emotion engine, it is possible to identify emotions from user speech and enable appropriate interaction.

[0801] The user installs and launches the application on a device such as a smartphone or tablet. This application has the function of acquiring audio signals through the microphone, and the acquired audio signals are processed in real time using noise cancellation technology. The device then amplifies these audio signals, providing the user with clear audio.

[0802] The audio signal is then sent to a server in the cloud, where it is converted into text using speech recognition and natural language processing technologies. During this process, an emotion engine analyzes the user's emotional state from the audio and extracts emotion data. This data indicates the user's current emotional state, and the analysis results are sent to the device.

[0803] The device dynamically adjusts the format of the text it displays based on acquired emotional data, for example, by changing the color and font size according to the user's emotions. Furthermore, when the device detects user stress or anxiety, it can automatically provide voice guidance for relaxation.

[0804] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can hear clear audio through the headset while reducing ambient noise, and simultaneously receive visual feedback in the form of text. Furthermore, if the emotion engine detects that the user is experiencing stress, the device can suggest playing relaxation music. In this way, the present invention provides a system that reduces the user's mental burden and supports smooth communication.

[0805] The following describes the processing flow.

[0806] Step 1:

[0807] The user launches the application on their device and selects voice input mode. This puts the device into a state where it can begin capturing audio.

[0808] Step 2:

[0809] The device uses its built-in microphone to acquire audio signals in real time. During this process, noise cancellation is enabled to remove ambient noise and obtain clear audio data.

[0810] Step 3:

[0811] The device analyzes the acquired audio signal and amplifies it to an appropriate volume for output. The amplified audio is then delivered to the user through earphones or speakers.

[0812] Step 4:

[0813] The device sends the de-noised audio signal to a server in the cloud, requesting data processing for speech recognition and sentiment analysis.

[0814] Step 5:

[0815] The server converts the received audio data into text using speech recognition technology. Simultaneously, it uses an emotion engine to analyze the user's emotional state from their voice and generates the results.

[0816] Step 6:

[0817] The server sends the analyzed text data and sentiment data back to the terminal. Communication is performed with low latency, prioritizing user convenience.

[0818] Step 7:

[0819] When a device displays received text on the screen, it automatically adjusts the formatting based on sentiment data. This includes changing the text color and font size according to the sentiment.

[0820] Step 8:

[0821] If the device detects negative emotions such as stress or anxiety, it will automatically provide voice guidance or music for relaxation. For example, it can play soothing music to help alleviate mental stress.

[0822] Step 9:

[0823] Users can adjust various settings options provided on their device as needed to enjoy a personalized experience. Other background sounds and responses that respond to specific emotions can also be configured.

[0824] (Example 2)

[0825] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0826] In modern society, users are required to handle audio information in various environments and understand the speaker's emotions within those contexts. However, conventional systems are susceptible to noise, making it difficult to accurately analyze emotional states. Furthermore, they lacked the functionality to provide appropriate feedback in response to the user's emotions. This invention aims to solve these problems and realize emotion-responsive interaction.

[0827] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0828] In this invention, the server includes means for acquiring audio information, means for amplifying the audio information, means for converting the amplified audio information into text information in real time, means for analyzing emotional information, and means for dynamically adjusting the display format based on the emotional information. This makes it possible to accurately analyze the emotional state of the user's speech and provide appropriate feedback while removing noise from the audio signal.

[0829] "Audio information" refers to sound waves represented as digital or analog signals.

[0830] "Means" refers to the equipment, apparatus, or method used to achieve a particular purpose.

[0831] "Amplification" refers to the process of increasing the strength or magnitude of a signal, making it clearer.

[0832] "Real-time" refers to a process that proceeds almost instantly, with virtually no time delay.

[0833] "Textual information" refers to a set of symbols and characters used to visually represent language.

[0834] "Display format" refers to the design and style that indicates how information and data are visually arranged.

[0835] "Dynamic adjustment" refers to the automatic modification or correction of information in response to changes in circumstances or conditions.

[0836] "Analysis" is the process of breaking down complex information or data into an easily understandable form and revealing its meaning and structure.

[0837] This invention is a system that enhances user interaction by acquiring audio information, amplifying it, converting it into text information, and further performing emotion analysis. Specifically, it is implemented as follows.

[0838] The user installs a dedicated application on their mobile device and starts the system by launching this application. This application uses the device's microphone to acquire voice information and utilizes noise cancellation technology to eliminate ambient noise, thus maintaining a clear voice signal.

[0839] Next, the device amplifies the acquired audio signal in real time, providing the user with clearer audio. This process ensures that the audio signal information is not lost, allowing the next step to be completed without any loss of quality.

[0840] The voice signal is transmitted via the internet to a server in the cloud. The server uses speech recognition technology to convert the voice signal into text information. Furthermore, a generative AI model is used to analyze and extract user emotional information from the voice. The information obtained from this emotional analysis is used to reflect and predict the user's psychological state.

[0841] The terminal dynamically adjusts the format of the displayed text information based on emotional information received from the server. For example, if the system determines that the user is calm, it can change the color scheme to a gentle blue, while if the user is excited, it can change to a red color scheme that indicates energy. This allows the system to provide a more user-friendly display.

[0842] As a concrete example, consider a scenario where a user uses this system during their commute. In this case, the user can use a headset to reduce ambient noise, hear clear audio, and simultaneously view visual text information on their device screen. Furthermore, if the user experiences stress, the system can provide relaxation guidance, creating a pleasant experience.

[0843] An example of a prompt statement would be, "I'm feeling stressed during my commute; please suggest some relaxation music." In this way, the present invention provides an effective method for understanding and promoting a user's emotions.

[0844] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0845] Step 1:

[0846] The user launches the application on their mobile device. As input, the user speaks into the device's microphone. The device acquires this audio signal and uses noise-canceling technology to remove unwanted ambient noise. The output is clear audio data.

[0847] Step 2:

[0848] Based on the acquired audio data, the device amplifies the audio signal, providing clear audio to the user in real time. The input is a noise-free audio signal, and the output is amplified audio. This amplification ensures that the user receives all the audio details clearly.

[0849] Step 3:

[0850] The terminal transmits amplified audio data to the server via the internet. The amplified audio signal, as input, becomes the data to be transmitted. The output is confirmation data after transmission is complete. This prepares the server to begin audio analysis.

[0851] Step 4:

[0852] The server uses speech recognition technology with the received audio data to convert speech into text. The input is the transmitted audio data, and the output is text data. The speech recognition engine records the spoken content as text.

[0853] Step 5:

[0854] The server uses a generative AI model to analyze user emotional information from text data. The input is text data, and the output is emotional information. The emotion engine identifies the user's psychological state and extracts the necessary emotional parameters.

[0855] Step 6:

[0856] Based on the emotional information, the server sends the analysis results to the terminal. The input is the analyzed emotional information, and the output is the emotional parameters sent to the terminal. This allows the terminal to prepare to display a response to the user.

[0857] Step 7:

[0858] The terminal dynamically adjusts the format of the text information displayed on the screen using the received emotion information. The input is emotion parameters received from the server, and the output is text information in the adjusted display format. Specifically, it changes the font color and size to display information according to the emotion.

[0859] Step 8:

[0860] Based on emotional information, the device detects user stress and anxiety and provides relaxation-enhancing audio guidance. Input is emotional information and the detection result of the user's stress level; output is playback of relaxation audio. This reduces the user's mental burden.

[0861] (Application Example 2)

[0862] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0863] A challenge exists in that when elderly people communicate with others, they may not receive appropriate responses that address their feelings, potentially increasing their stress and anxiety. While efficient and emotionally responsive support is needed in care settings, current technology has made it difficult to fully achieve this.

[0864] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0865] In this invention, the server includes a device for acquiring audio signals, a processing device for analyzing emotions, and a device for providing visual displays corresponding to the user's emotions. This makes it possible to analyze the emotions of elderly people and provide visual displays and audio guidance corresponding to those emotions.

[0866] A "speech signal" is a representation of sound as an electrical signal, and typically includes the content of human speech.

[0867] An "device" is a machine or system designed to achieve a specific function or purpose.

[0868] "Amplification" is the process of increasing the signal strength, and it is a process performed to make audio signals clearer.

[0869] "Real-time" refers to a state where processing is performed instantly without delay.

[0870] A "text" is a linguistic expression that uses words arranged according to certain rules to convey a specific meaning.

[0871] "Emotion" is a psychological state of a human being, and includes feelings such as joy and sadness.

[0872] "Analysis" is the process of breaking down an object into its constituent parts and understanding its properties.

[0873] A "processing device" is a device that manipulates or converts given data or signals according to instructions.

[0874] "Visual representation" means showing information in a way that is visible to the eye.

[0875] This invention is a system that allows users to receive real-time emotion analysis via voice signals and receive visual feedback and relaxation guidance. First, the terminal acquires the user's voice signal through a microphone. This terminal incorporates noise-canceling technology, which removes external noise and clarifies the voice signal. The voice signal is amplified and then transmitted to a cloud server.

[0876] When the server receives an audio signal, it uses speech recognition software to convert the signal into text. Next, the server uses an emotion analysis engine to identify the user's emotional state based on this text. This analysis may involve third-party speech recognition platforms or natural language processing technologies (e.g., OpenAI's GPT model).

[0877] The results of the emotion analysis are returned to the user's device, which then displays them visually. Furthermore, if stress or anxiety is detected, the device provides relaxation-enhancing audio guidance or music. This allows for appropriate responses tailored to the user's emotional state.

[0878] As a concrete example, consider a situation where an elderly person is conversing with a caregiver in a care facility. Using this system, the elderly person's voice is instantly converted into text, and their emotional state is analyzed. For example, if the emotional analysis determines that the elderly person is feeling anxious, the device can display a visually reassuring message and play relaxing music. This system allows caregivers to respond immediately to the elderly person's mental state.

[0879] An example of a prompt is, "Using this system, how would you like to support the emotional changes of elderly individuals during their daily communication?" Based on this prompt, the generative AI model can construct appropriate interactions that respond to their emotions.

[0880] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0881] Step 1:

[0882] The device acquires audio signals using a microphone. The input is the user's speech, and the output is digitized audio data. This data is processed into a clear audio signal by removing noise using noise cancellation technology.

[0883] Step 2:

[0884] The terminal sends a clear audio signal to the server. The input is a noise-free audio signal, and the output is the signal transfer to the cloud server. The server receives this signal and prepares for the next processing step.

[0885] Step 3:

[0886] The server uses a speech recognition engine to convert the audio signal into text. The input is the transmitted audio signal, and the output is the text data corresponding to that audio. This text is generated using technologies such as generative AI models.

[0887] Step 4:

[0888] The server uses an emotion analysis engine to identify the user's emotional state based on the converted text. The input is text data, and the output is emotion data. The server utilizes natural language processing techniques to analyze emotions.

[0889] Step 5:

[0890] The server instructs the system to generate visual feedback corresponding to the user's emotional state, and, if necessary, relaxation sounds. The input is emotional data, and the output is the instructed feedback content. Prompts may also be used in this process.

[0891] Step 6:

[0892] The terminal provides the user with visual displays and audio guidance based on feedback instructions from the server. The input is the instructions from the server, and the output is the user-viewable screen display and the audible relaxation audio.

[0893] Step 7:

[0894] The user follows the provided feedback, confirms the instructed actions, and performs relaxation as needed. Input is visual and auditory feedback, and output is the user's response or change in internal feelings.

[0895] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0896] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0897] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0898] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0899] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0900] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0901] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0902] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0903] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0904] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0905] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0906] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0907] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0908] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0909] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0910] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0911] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0912] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0913] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0914] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0915] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0916] The following is further disclosed regarding the embodiments described above.

[0917] (Claim 1)

[0918] An acquisition means for acquiring audio signals,

[0919] Amplification means for amplifying the aforementioned audio signal,

[0920] A conversion means that converts the amplified audio signal into text in real time,

[0921] Display means for displaying the converted text,

[0922] A system that includes this.

[0923] (Claim 2)

[0924] The system according to claim 1, further comprising a notification means for notifying the user when an anomaly is detected.

[0925] (Claim 3)

[0926] The system according to claim 1, wherein the audio signal is processed to remove noise.

[0927] "Example 1"

[0928] (Claim 1)

[0929] A device equipped with the function of acquiring audio data,

[0930] A device for reducing background noise from the aforementioned audio data,

[0931] A device for amplifying the reduced audio data,

[0932] A device for transmitting the amplified audio data,

[0933] A device equipped with the function of analyzing the transmitted audio data on the cloud and converting it into text data,

[0934] A device having the function of receiving the aforementioned character data and displaying it visually,

[0935] An information processing system that includes this.

[0936] (Claim 2)

[0937] The information processing system according to claim 1, further comprising means for notifying a user when an anomaly is detected.

[0938] (Claim 3)

[0939] The information processing system according to claim 1, wherein the audio data is processed to reduce background noise.

[0940] "Application Example 1"

[0941] (Claim 1)

[0942] An acquisition means for acquiring audio signals,

[0943] Amplification means for amplifying the aforementioned audio signal,

[0944] A conversion means that converts the amplified audio signal into text in real time,

[0945] Display means for displaying the converted text,

[0946] Support measures to facilitate dialogue between care workers and the elderly,

[0947] A system that includes this.

[0948] (Claim 2)

[0949] The system according to claim 1, further comprising a notification means for notifying the user when an anomaly is detected.

[0950] (Claim 3)

[0951] The system according to claim 1, wherein the audio signal is processed to remove noise.

[0952] "Example 2 of combining an emotion engine"

[0953] (Claim 1)

[0954] Means for acquiring audio information,

[0955] means for amplifying the aforementioned audio information,

[0956] The means for converting the amplified audio information into text information in real time,

[0957] means for displaying the aforementioned character information,

[0958] A means for analyzing emotional information from the aforementioned audio information,

[0959] Means for dynamically adjusting the display format based on the aforementioned emotional information,

[0960] A system that includes this.

[0961] (Claim 2)

[0962] The system according to claim 1, further comprising means for notifying the user when an abnormality is detected.

[0963] (Claim 3)

[0964] The system according to claim 1, wherein the audio information is processed to remove noise.

[0965] "Application example 2 when combining with an emotional engine"

[0966] (Claim 1)

[0967] A device for acquiring audio signals,

[0968] A device for amplifying the aforementioned audio signal,

[0969] A device that converts the amplified audio signal into text in real time,

[0970] A device for displaying the converted text,

[0971] A processing unit that analyzes emotions,

[0972] A device that provides visual displays that respond to the user's emotions,

[0973] A system that includes this.

[0974] (Claim 2)

[0975] The system according to claim 1, which notifies the user when it detects an abnormality or stress and provides voice guidance for relaxation.

[0976] (Claim 3)

[0977] The system according to claim 1, wherein the audio signal is processed to remove noise. [Explanation of symbols]

[0978] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An acquisition means for acquiring audio signals, Amplification means for amplifying the aforementioned audio signal, A conversion means that converts the amplified audio signal into text in real time, Display means for displaying the converted text, Support measures to facilitate dialogue between care workers and the elderly, A system that includes this.

2. The system according to claim 1, further comprising a notification means for notifying the user when an abnormality is detected.

3. The system according to claim 1, wherein the audio signal is processed to remove noise.

Images