System

Abstract

To provide a system for transmitting a voice inputted from a terminal to a server, converting the voice into characters in real time and outputting the characters to the terminal.SOLUTION: A system comprising: means for receiving voice input; means for capturing the voice input as voice data; means for transmitting the voice data to a server; means for converting the voice data to text data; means for returning the text data to a terminal; and means for displaying the text data to a user.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional face-to-face or telephone communication, there is a lack of a way to confirm audio as text, making it difficult for people with hearing impairments or in noisy environments.In addition, there is a need for a way to improve the quality of communication by transcribing the content of the conversation in real time and visually checking it, as it is not possible to confirm or record the content. [Means for solving the problem]

[0005] This invention includes a means for accepting voice input, capturing it as voice data, and transmitting it to a server. It also includes a means for the server to convert the voice data into text data and return it to the terminal. It also includes a means for the terminal to display the text data to the user. Specifically, by using AR glasses to capture face-to-face conversations and a smartphone to capture phone conversations and convert them into text data, which are then visually displayed in real time, it enables smooth communication even for the hearing impaired or in noisy environments.

[0006] "Voice input" refers to the act of collecting the user's voice as digital data through the terminal's microphone.

[0007] "Audio Data" means captured audio data stored and processed in digital form.

[0008] "Capture" is the process of collecting and recording audio and video as digital data.

[0009] A "server" is a centrally managed computing resource, a device for processing and storing voice data and providing various services.

[0010] "Character data" is voice data that has been analyzed and converted into corresponding text format data.

[0011] "Conversion" is the process of changing data from one form to another.

[0012] A "terminal" is a device that a user directly operates to input and output information, including smartphones and AR glasses.

[0013] "Return" is the process of sending back the processing results from the server to the terminal.

[0014] "Display" refers to the act of visually showing character data on a terminal display.

[0015] "AR glasses" are glasses-type devices that use Augmented Reality technology to overlay digital information onto the user's field of vision.

[0016] A "smartphone" is a portable device that combines communication and computing capabilities. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention relates to a system for displaying voice communication as text in real time. The system converts voice into text and allows users to visually confirm the text when communicating with a conversation partner in person or over the phone.

[0039] In a face-to-face conversation

[0040] 1. The user puts on the AR glasses and talks to the other person.

[0041] The AR glasses have built-in microphones that capture audio as users speak.

[0042] 2. The device (AR glasses) captures audio in real time.

[0043] The voices of the person you are speaking to are collected as digital voice data.

[0044] 3. The device sends the collected voice data to the server.

[0045] The communication function of the AR glasses is used to send audio data to the server.

[0046] 4. The server analyzes the voice data and converts it into text data.

[0047] The server uses voice recognition technology such as Whisper to convert the audio into text.

[0048] 5. The server returns the converted character data to the terminal.

[0049] The text data is then sent back to the AR glasses.

[0050] 6. The device displays the received text data on the AR display.

[0051] Users can view the content of the conversation in real time as text.

[0052] Examples:

[0053] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the AR glasses, where "What time is the next meeting?" is displayed on the display. The user can visually confirm the content.

[0054] For telephone communication

[0055] 1. The user launches the calling app on their smartphone and makes a call.

[0056] The smartphone will begin recording the audio of both parties during the call.

[0057] 2. The device (smartphone) captures the audio from both parties.

[0058] The audio of the call is collected as digital data.

[0059] 3. The device sends the captured audio data to the server.

[0060] The smartphone encrypts the voice data it collects and sends it to a server.

[0061] 4. The server analyzes the voice data and converts it into text data.

[0062] The dialogue is converted into text using voice recognition technology such as the Whisper model.

[0063] 5. The server returns the converted character data to the terminal.

[0064] The text data is sent to the smartphone.

[0065] 6. The device displays the received text data on the smartphone screen.

[0066] Users can view the contents of phone conversations as text in real time.

[0067] Examples:

[0068] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server converts the audio into text and sends it back to the smartphone, where "Tell me about your new project" is displayed on the screen. The user can visually confirm the content.

[0069] This system converts speech into text in real time during face-to-face and remote communication, enabling smooth conversations even for people with hearing impairments or in noisy environments. This allows users to visually check the transcribed data, improving the quality of communication.

[0070] The processing flow will be explained below.

[0071] In a face-to-face conversation

[0072] Step 1:

[0073] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[0074] Step 2:

[0075] The user's voice is captured in real time by the microphone in the AR glasses.

[0076] Step 3:

[0077] The device (AR glasses) temporarily stores the captured audio data and compresses and encrypts it.

[0078] Step 4:

[0079] The device sends compressed and encrypted audio data to the server using a secure protocol.

[0080] Step 5:

[0081] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0082] Step 6:

[0083] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0084] Step 7:

[0085] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[0086] Step 8:

[0087] The device (AR glasses) decodes the received text data and displays it on the AR glasses' display.

[0088] Step 9:

[0089] Users can see what their conversation partner is saying in real time.

[0090] For telephone communication

[0091] Step 1:

[0092] The user launches the calling app on their smartphone and calls the person they want to call.

[0093] Step 2:

[0094] The user's voice and the voice of the other party are captured in real time by the smartphone's microphone.

[0095] Step 3:

[0096] The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[0097] Step 4:

[0098] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[0099] Step 5:

[0100] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0101] Step 6:

[0102] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0103] Step 7:

[0104] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[0105] Step 8:

[0106] The device (smartphone) decodes the received text data and displays it on the call app screen.

[0107] Step 9:

[0108] Users can view the contents of the call in real time as text.

[0109] This system allows face-to-face and telephone communications to be visually confirmed as text through a series of processes that captures voice in real time, converts it into text, and displays it.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] Conventional voice communication has the drawback of being difficult to communicate smoothly when the speaker is inaudible or when used in environments with a lot of background noise. Furthermore, people with hearing impairments and some elderly people have difficulty understanding spoken information, creating a need for a means to convert speech into text in real time. Furthermore, from the perspective of privacy protection, encryption technology is also needed to process voice data securely.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into text data, means for returning the text data to the terminal, means for displaying the text data to the user, means for the means to use voice recognition technology to generate, transmit, and display visually identifiable text data in real time, and means for encrypting the voice data transmission and reception process. This allows users to visually confirm voice as text in real time, enabling smooth communication even in environments with a lot of background noise or for the hearing impaired. Furthermore, encryption of the voice data also enhances privacy protection.

[0115] The "means for accepting voice input" refers to a device or software function that captures the voice uttered by the user and environmental sounds and acquires the data necessary for subsequent processing.

[0116] "Means for capturing audio data" refers to the functionality of a device or software that digitally records audio input and converts it into a format suitable for further processing.

[0117] The "means for transmitting audio data to a server" refers to a device or software function that transfers captured audio data to a server via a communication line.

[0118] "Means for converting voice data into text data" refers to the functionality of a device or software within the server that converts voice data into text using voice recognition technology.

[0119] "Means for returning character data to the terminal" refers to the function of a device or software that sends the text data converted by the server back to the terminal via a communication line.

[0120] The "means for displaying character data to the user" refers to a device or software function that visually presents the returned text data to the user on a terminal display or the like.

[0121] "Means of using voice recognition technology" refers to technology or software functions for analyzing voice data and converting it into text with high accuracy.

[0122] "Encryption means" refers to the function of a device or software that encrypts data to maintain confidentiality during the communication of voice data and text data.

[0123] "Augmented reality glasses" are glasses-type devices that can be worn by the user to overlay virtual information (such as text data) onto the real world.

[0124] A "mobile phone" is a small portable communication terminal capable of voice calls and data communications.

[0125] This invention relates to a system that displays voice communication as text in real time, and is a system that allows a user to convert voice into text and visually confirm it when communicating with a conversation partner in person or over the phone.

[0126] This system is implemented as follows: First, a user uses either augmented reality glasses or a mobile phone. When a user wants to have a face-to-face conversation, the user puts on the augmented reality glasses and starts the conversation. The augmented reality glasses have a built-in microphone that captures the voice of the person they are talking to in real time. When a user wants to have a conversation over the phone, the voice data is captured using the mobile phone's calling app.

[0127] The device (augmented reality glasses or mobile phone) encodes the captured voice data and transmits it to a server via a communication line. The server receives the transmitted voice data and converts it into text data using voice recognition technology. Specifically, a voice recognition model such as Whisper by OpenAI (registered trademark) is used.

[0128] Once the voice has been converted into text data, the server re-encrypts the text data and sends it back to the terminal via the communication line. The terminal then decrypts the received text data and displays it on a display so that the user can visually confirm it. This process occurs in real time, so the user can immediately see the contents of the conversation in text form.

[0129] The system also uses encryption technology to protect the privacy of users' voice data and converted text data, allowing users to use the system with peace of mind.

[0130] Specific examples

[0131] In a face-to-face conversation: When a user says, "What time is the next meeting?", the augmented reality glasses capture the voice and send it to the server. The server converts the voice into text data and sends it back to the augmented reality glasses. The user can see the text "What time is the next meeting?" displayed on the display inside the AR glasses.

[0132] In the case of telephone communication: When a user says "Tell me about your new project" on the phone, the mobile phone captures this voice and sends it to the server. The server converts the voice into text data and sends it back to the mobile phone. The user can see the text "Tell me about your new project" on the mobile phone screen.

[0133] Prompt Sentence Examples

[0134] 1. "What time is the next meeting?"

[0135] 2. "Tell me about your new project."

[0136] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0137] Step 1:

[0138] The device (augmented reality glasses or mobile phone) accepts voice input.

[0139] Input: User's voice.

[0140] How it works: In the case of augmented reality glasses, a microphone picks up what the user says and stores it as digital audio data. In the case of a mobile phone, the voice is recorded through a calling app.

[0141] Output: Captured as digital audio data and temporarily stored in memory.

[0142] Step 2:

[0143] The device sends the captured audio data to the server.

[0144] Input: Digital audio data.

[0145] What happens: Your device uses your internet connection to encrypt and send audio data to a server. Wi-Fi or cellular data is used.

[0146] Output: The encrypted audio data arrives at the server.

[0147] Step 3:

[0148] The server receives the audio data and begins analyzing it.

[0149] Input: Encrypted audio data.

[0150] Specific operation: The server decodes the received voice data and converts it into text using voice recognition technology (e.g., OpenAI's Whisper).

[0151] Output: The parsed text data is generated.

[0152] Step 4:

[0153] The server returns the converted character data to the terminal.

[0154] Input: Parsed text data.

[0155] Specific operation: The server encrypts the text data and sends it to the device over the communication line, again using Wi-Fi or mobile data.

[0156] Output: Encrypted text data arrives at the terminal.

[0157] Step 5:

[0158] The terminal decodes the received character data and displays it to the user.

[0159] Input: Encrypted text data.

[0160] Specific operation: The device decodes the received text data and displays it on the display in the case of augmented reality glasses, or on the screen in the case of a mobile phone.

[0161] Output: Text data that can be visually confirmed by the user.

[0162] Specific examples

[0163] Prompt Sentence Examples

[0164] 1. "What time is the next meeting?"

[0165] 2. "Tell me about your new project."

[0166] (Application example 1)

[0167] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0168] Currently, it is difficult for hearing-impaired employees to communicate with customers in noisy brick-and-mortar stores. It is also difficult for employees to clearly hear what customers are saying when they are wearing masks or in crowded environments. In these environments, an effective method for smoothly conducting conversations with customers is needed.

[0169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0170] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into character data, means for returning the character data to the terminal, means for displaying the character data on a visual information display device, and means for the visual information display device to be used by employees of the physical store to communicate with customers. This allows employees of the physical store to visually confirm what customers are saying, enabling smoother communication.

[0171] A "means for accepting voice input" is a device or part of a system that collects voice data, recognizes voice through hardware such as a microphone, and processes it as digital data.

[0172] "Means for capturing voice input as audio data" refers to a device or process that records received audio in a digital data format and converts it into analyzable data.

[0173] "Means for transmitting audio data to a server" refers to a communication interface or protocol for transferring captured audio data to a remote server via a network.

[0174] "Means for converting voice data into text data" refers to the speech recognition technology or software used to analyze received voice data and generate a corresponding string of characters.

[0175] "Means for returning character data to the terminal" refers to the communication functions and protocols for returning character data generated by the server to the original terminal via the network.

[0176] "Means for displaying character data on a visual information display device" refers to a display device such as a display or AR glasses, and its control system, for visually presenting the character data returned to the terminal to the user.

[0177] A "visual information display device" is an electronic device that provides information visually to a user, and includes displays, AR glasses, head-mounted displays, etc.

[0178] A "physical store" is a physical space, a place where products and services are offered and where store staff and customers can communicate face-to-face.

[0179] A "personal digital assistant" is a portable electronic device capable of communicating voice and data, including smartphones and tablets.

[0180] This invention provides a system that converts voice input into text and displays it in real time when employees of a brick-and-mortar store communicate with customers. This system accepts voice input, converts it into text data, and displays it to employees on a visual information display device, enabling appropriate communication even in noisy environments or for those with hearing impairments.

[0181] Hardware and software used

[0182] This system requires the following hardware and software:

[0183] Hardware:

[0184] A microphone that accepts voice input (built into the visual information display device)

[0185] Visual information display devices (e.g., AR glasses, smart glasses)

[0186] Mobile information terminals (e.g. smartphones)

[0187] software:

[0188] Audio Capture Software

[0189] Communication Interface

[0190] Speech recognition engine (e.g. Whisper model)

[0191] Data Display Software

[0192] The server converts the voice data into text data using voice recognition technology, and the terminal receives the text data and displays it on a visual information display device.

[0193] Data processing and calculation methods

[0194] 1. Audio capture:

[0195] The user accepts voice input through the visual information display device. For example, a store employee wears smart glasses and converses with a customer.

[0196] 2. Sending audio data:

[0197] The device then sends the captured audio data to the server, where it encrypts the data using its communication function to ensure secure transmission.

[0198] 3. Speech recognition and conversion to text data:

[0199] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into corresponding text data.

[0200] 4. Returning character data:

[0201] The server returns the character data to the terminal, which displays the character data on a visual information display device.

[0202] 5. Character display:

[0203] Users can check text data in real time through the visual information display device, which allows them to visually recognize what the customer is saying and respond appropriately.

[0204] Specific examples

[0205] For example, if a customer says to a store employee, "Please explain this product to me," the smart glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the smart glasses as "Please explain this product to me." As a result, the store employee can see the text displayed on the visual information display device and understand the content of the conversation.

[0206] Example prompt for a generative AI model:

[0207] "Please tell me the specific steps to implement a system that transcribes conversations in a physical store in real time. Please provide a detailed explanation including voice capture, sending the voice data to the server, voice recognition, and displaying the text data."

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1:

[0210] A user wears a visual information display device (e.g., smart glasses) and talks with a customer.

[0211] Input: Customer voice

[0212] How it works: A microphone built into the visual display captures sound.

[0213] Output: Digital audio data

[0214] Step 2:

[0215] The terminal transmits the captured audio data to a server via a network.

[0216] Input: Digital audio data

[0217] How it works: The device's communications interface encrypts the audio data and sends it to the server.

[0218] Output: Audio data sent to the server

[0219] Step 3:

[0220] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into text data.

[0221] Input: Audio data transferred to the server

[0222] How it works: The server's speech recognition engine analyzes the audio data and generates corresponding text data.

[0223] Output: Character data

[0224] Step 4:

[0225] The server returns the generated character data to the terminal.

[0226] Input: Character data

[0227] Operation: The server's communication interface sends character data to the terminal.

[0228] Output: Character data sent to the terminal

[0229] Step 5:

[0230] The terminal displays the received character data on a visual information display device in real time.

[0231] Input: Character data sent to the terminal

[0232] Operation: The visual information display device displays the received character data on the display.

[0233] Output: Text data that can be visually confirmed by the user

[0234] Specific examples

[0235] Let's consider what happens when a customer asks an employee in a physical store, "Please explain this product to me."

[0236] Step 1: The microphone built into the visual display captures the audio, "Please describe this product."

[0237] Step 2: The device's communication interface encrypts this voice data and sends it to the server.

[0238] Step 3: The server's speech recognition engine (e.g., Whisper model) analyzes the voice data and generates text data such as "Please describe this product."

[0239] Step 4: The server's communication interface sends the character data to the terminal.

[0240] Step 5: The visual information display device displays the received text data on the screen, allowing the employee to confirm the content as text.

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] This invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts voice into text and analyzes and visually confirms the user's emotions.

[0243] In a face-to-face conversation

[0244] 1. The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[0245] 2. The device (AR glasses) captures the voice in real time. The voice of the person you are talking to is collected as digital audio data.

[0246] 3. The device temporarily stores the collected voice data and compresses and encrypts it.

[0247] 4. The device sends the compressed and encrypted audio data to the server. This communication uses a secure protocol.

[0248] 5. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0249] 6. The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0250] 7. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[0251] 8. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the terminal.

[0252] 9. The device (AR glasses) decodes the received text data and emotion data and displays it on the AR glasses display.

[0253] 10. Users can check the content of the conversation as well as the emotional state of the person they are talking to in real time.

[0254] Examples:

[0255] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server transcribes the voice, and at the same time, the emotion recognition engine analyzes whether the user is speaking in a serious tone, and the message "What time is the next meeting? (Emotion: Serious)" is displayed on the AR glasses' display. The user can visually confirm the content of the conversation and the emotion.

[0256] For telephone communication

[0257] 1. A user launches a calling app on their smartphone and makes a call to the other party. The user's voice and the other party's voice are captured in real time by the smartphone's microphone.

[0258] 2. The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[0259] 3. The device sends the compressed and encrypted audio data to the server, again using a secure protocol.

[0260] 4. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0261] 5. The server analyzes the voice data and converts it into text data, which is then stored in internal memory.

[0262] 6. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[0263] 7. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the device.

[0264] 8. The device (smartphone) decodes the received text data and emotion data and displays it on the call app screen.

[0265] 9. Users can check the content of the call as well as the emotional state of the other party in real time.

[0266] Examples:

[0267] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server transcribes the audio, and at the same time, an emotion recognition engine analyzes whether the user is speaking in an interested tone, and the message "Tell me about your new project (emotion: interest)" is displayed on the smartphone screen. The user can visually confirm the content of the call and the emotion.

[0268] This system improves the quality of face-to-face and telephone communication by capturing speech in real time, transcribing it, analyzing emotions, and displaying the results simultaneously. This process allows users to visually confirm both the content and emotions of the communication, enabling smoother dialogue.

[0269] The processing flow will be explained below.

[0270] In a face-to-face conversation

[0271] Step 1:

[0272] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[0273] Step 2:

[0274] The device (AR glasses) captures audio in real time, and the voice of the person you are talking to is collected as digital audio data.

[0275] Step 3:

[0276] The device temporarily stores the collected voice data in its internal memory and simultaneously compresses and encrypts it.

[0277] Step 4:

[0278] The device sends compressed and encrypted audio data to the server using a secure protocol.

[0279] Step 5:

[0280] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[0281] Step 6:

[0282] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0283] Step 7:

[0284] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[0285] Step 8:

[0286] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[0287] Step 9:

[0288] The device decodes the received text data and emotion data and displays it on the AR glasses' display.

[0289] Step 10:

[0290] Users can check the content of the conversation as well as the emotions of their conversation partner in real time.

[0291] For telephone communication

[0292] Step 1:

[0293] A user starts a calling app on their smartphone and makes a call to the other party. The voices of both parties are captured in real time by the smartphone's microphone during the call.

[0294] Step 2:

[0295] The device (smartphone) temporarily stores the captured audio data in its internal memory, and simultaneously compresses and encrypts it.

[0296] Step 3:

[0297] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[0298] Step 4:

[0299] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[0300] Step 5:

[0301] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0302] Step 6:

[0303] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[0304] Step 7:

[0305] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[0306] Step 8:

[0307] The device decodes the received text data and emotion data and displays them on the call app screen.

[0308] Step 9:

[0309] Users can see the content of the call as well as the emotions of the person they are talking to in real time.

[0310] In this way, a system is realized that converts voice communication into text and then analyzes and displays emotional data, improving the quality of face-to-face and telephone communication.

[0311] Example 2

[0312] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0313] In modern voice communication, understanding the emotions of the person you're speaking to is extremely important. However, conventional systems only transcribe speech and are unable to analyze emotions, making it difficult for users to fully understand the content of the conversation. Furthermore, there was a lack of technology that could consistently perform real-time transcription and emotion analysis, preventing efficient communication support. This could result in a decline in the quality of the conversation and the risk of misunderstandings and breakdowns in communication.

[0314] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0315] In this invention, the server includes means for decoding voice data and converting it into text data, means for analyzing emotions from the voice data, and means for compressing and encrypting the text data and emotional data, thereby enabling real-time transcription and emotional analysis of voice communication.

[0316] "Voice input" refers to words or dialogue spoken by a user.

[0317] "Audio Data" means captured audio input stored in digital form.

[0318] "Compression" is the technique of reducing the size of data so that it can be stored and transmitted efficiently.

[0319] "Encryption" is the process of converting data into an unintelligible form for protection.

[0320] A "server" is a computer system responsible for analyzing and processing audio data.

[0321] "Decoding" is the technique of restoring encrypted data to its original form.

[0322] "Character data" is data in text format into which voice data has been converted.

[0323] "Emotion analysis" is a technology that identifies a speaker's emotions by analyzing the tone and pitch of a voice.

[0324] A "terminal" is a device that captures and displays audio data.

[0325] "Display" refers to the act of visually showing text and emotion data to the user.

[0326] A "head-mounted display" is a display device that is worn on the head.

[0327] A "personal digital assistant" is a portable device that captures and displays audio data.

[0328] The present invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts the voice into text and analyzes and visually confirms the user's emotions.

[0329] This system consists of a terminal that accepts voice input and captures the voice data, and a server that analyzes the voice data. Specifically, the following hardware and software are used:

[0330] Hardware

[0331] 1. Terminal: A device that captures and displays audio data. For face-to-face conversations, a head-mounted display is used, and for telephone communication, a personal digital assistant is used.

[0332] 2. Server: A computer system that decodes audio data, converts it to text data, and analyzes emotions.

[0333] software

[0334] 1. Speech recognition software: converts voice data into text data. For example, we will use Whisper.

[0335] 2. Emotion recognition engine: Identifies emotions by analyzing voice tone, pitch, etc.

[0336] Processing flow

[0337] In a face-to-face conversation

[0338] 1. The user puts on the head-mounted display and prepares to start a conversation with the other person. The display's microphone prepares for voice capture.

[0339] 2. The device (head-mounted display) captures audio in real time and temporarily stores it as digital audio data.

[0340] 3. The device compresses and then encrypts the collected audio data.

[0341] 4. The device sends the audio data to the server using a secure communication protocol such as TLS or HTTPS.

[0342] 5. The server decodes the received voice data and inputs it into the voice recognition software.

[0343] 6. The server analyzes the voice data and converts it into text data.

[0344] 7. The server uses an emotion recognition engine to analyze emotions from the voice data.

[0345] 8. The server re-compresses and encrypts the converted text data and emotion data and sends them back to the terminal.

[0346] 9. The device (head-mounted display) decodes the voice and emotion data and displays it on the display.

[0347] 10. The user visually confirms the displayed text data and emotional information.

[0348] For telephone communication

[0349] 1. The user launches the calling app on their mobile device and calls the person they want to call. The microphone on the mobile device prepares to capture audio.

[0350] 2. The device (mobile information terminal) captures the audio data in real time and temporarily stores it.

[0351] 3. The device compresses and encrypts the captured audio data.

[0352] 4. The device sends the audio data to the server using a secure communication protocol.

[0353] 5. The server decodes the received data and inputs it into the voice recognition software.

[0354] 6. The server analyzes the voice data and converts it into text data.

[0355] 7. The server analyzes the voice data using an emotion recognition engine to identify the emotion.

[0356] 8. The server re-compresses the converted text data and emotion data, encrypts them, and returns them to the terminal.

[0357] 9. The device (mobile information terminal) decodes the data and displays it in the calling application.

[0358] 10. The user visually confirms the text data and emotional information.

[0359] Specific examples

[0360] As an example of a prompt sentence, if a user says "What time is the next meeting?", the AR glasses will capture the voice and send it to the server. The server will convert the voice into text and analyze it with an emotion recognition engine, which will then display "What time is the next meeting? (Emotion: Serious)" on the AR glasses. The user can see this display in real time.

[0361] Furthermore, in the case of telephone communication, when a user says, "Tell me about your new project," the mobile information terminal captures the voice and sends it to the server. The server converts it into text data, which is displayed on the mobile information terminal as, "Tell me about your new project (emotion: interest)."

[0362] This system can improve the quality of face-to-face and telephone communication by transcribing speech in real time, analyzing emotions, and displaying the results.

[0363] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0364] In a face-to-face conversation

[0365] Step 1:

[0366] The user puts on the head-mounted display and prepares to start a conversation with the other person. The microphone of the head-mounted display prepares to capture voice.

[0367] Input: The user puts on the head-mounted display and begins interacting.

[0368] Output: The microphone is now operational and ready to capture audio.

[0369] Step 2:

[0370] The device (head-mounted display) captures surrounding sounds in real time, and the microphone collects the voice of the person you are talking to with high accuracy and temporarily stores it as digital audio data.

[0371] Input: Voice of the person you are talking to.

[0372] Output: Temporarily stored as digital audio data.

[0373] Step 3:

[0374] The terminal compresses the collected audio data and then encrypts it.

[0375] Input: Digital audio data.

[0376] Output: Compressed and encrypted audio data.

[0377] Step 4:

[0378] The device sends compressed and encrypted audio data to the server using secure communication protocols such as TLS and HTTPS.

[0379] Input: Compressed and encrypted audio data.

[0380] Output: Sent to the server.

[0381] Step 5:

[0382] The server decodes the received audio data and restores it to the original audio data.

[0383] Input: Encrypted audio data.

[0384] Output: Decoded audio data.

[0385] Step 6:

[0386] The server inputs the recovered voice data into voice recognition software (e.g., Whisper) and converts it into text data, which is then temporarily stored in internal memory.

[0387] Input: Decoded audio data.

[0388] Output: Character data.

[0389] Step 7:

[0390] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes the tone, pitch and other parameters of the voice to identify emotions.

[0391] Input: Decoded audio data.

[0392] Output: Emotion data.

[0393] Step 8:

[0394] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[0395] Input: Text data and emotion data.

[0396] Output: Compressed and encrypted text and emotion data.

[0397] Step 9:

[0398] The terminal (head-mounted display) decodes the voice and emotion data and displays it on the display.

[0399] Input: Compressed and encrypted character data and emotion data.

[0400] Output: Decoded character data and emotion data are displayed on the screen.

[0401] Step 10:

[0402] The user visually checks the displayed character data and emotion information.

[0403] Input: Character data and emotion data displayed on the display.

[0404] Output: User confirmation.

[0405] For telephone communication

[0406] Step 1:

[0407] The user launches a calling application on the mobile information terminal and makes a call to the other party. The microphone on the mobile information terminal prepares for voice capture.

[0408] Input: The user launches a calling app on a mobile information terminal and starts a call.

[0409] Output: The microphone is now operational and ready to capture audio.

[0410] Step 2:

[0411] The device (mobile information terminal) captures the voices of the user and the other party in real time, and temporarily stores them as digital voice data.

[0412] Input: Caller and user voice.

[0413] Output: Temporarily stored as digital audio data.

[0414] Step 3:

[0415] The device compresses the captured audio data and then encrypts it.

[0416] Input: Digital audio data.

[0417] Output: Compressed and encrypted audio data.

[0418] Step 4:

[0419] The device sends compressed and encrypted audio data to the server using a secure communication protocol.

[0420] Input: Compressed and encrypted audio data.

[0421] Output: Sent to the server.

[0422] Step 5:

[0423] The server decodes the received audio data and restores it to the original audio data.

[0424] Input: Encrypted audio data.

[0425] Output: Decoded audio data.

[0426] Step 6:

[0427] Similarly, the server inputs the voice data into voice recognition software and converts it into text data, which is then temporarily stored in internal memory.

[0428] Input: Decoded audio data.

[0429] Output: Character data.

[0430] Step 7:

[0431] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes parameters such as tone, pitch, speed, and volume of the voice to identify emotions.

[0432] Input: Decoded audio data.

[0433] Output: Emotion data.

[0434] Step 8:

[0435] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[0436] Input: Text data and emotion data.

[0437] Output: Compressed and encrypted text and emotion data.

[0438] Step 9:

[0439] The terminal (mobile information terminal) decodes the received text data and emotion data and displays them on the screen of the calling app.

[0440] Input: Compressed and encrypted character data and emotion data.

[0441] Output: Decoded character data and emotion data are displayed on the screen.

[0442] Step 10:

[0443] The user visually checks the displayed character data and emotion information.

[0444] Input: Character data and emotion data displayed on the screen.

[0445] Output: User confirmation.

[0446] (Application example 2)

[0447] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0448] In today's brick-and-mortar stores, it is difficult to accurately grasp a customer's intentions and emotions when dealing with customers, making improving customer satisfaction a challenge. If it were possible to instantly understand a customer's emotions on-site, more appropriate responses could be made. Furthermore, displaying voice communication as text in real time would clearly understand the customer's intentions and reduce misunderstandings. However, existing systems that achieve these goals are limited, and in many cases, problems include insufficient emotion analysis capabilities and insufficient system response speed.

[0449] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0450] In this invention, the server includes means for receiving voice input in real time, means for converting voice data into text data using voice recognition software, means for analyzing emotions from the voice data, and means for returning the analyzed emotional data and text data to the terminal. This makes it possible to convert conversations with customers into text in real time and to visually display the customer's emotions. This allows store clerks to instantly understand the customer's emotions and take appropriate action, contributing to improved customer satisfaction.

[0451] "Voice input" is the means by which human speech is received in digital form.

[0452] "Audio data" is data that represents audio as a digital signal.

[0453] A "server" is a computer system that processes information and manages data on a network.

[0454] "Character data" refers to text data converted from voice data using voice recognition technology.

[0455] "Terminal" means a device with input and output capabilities that can be directly operated by a user.

[0456] A "user" is a person who uses the system.

[0457] "Visual device" refers to a device that has the function of displaying visual information, and includes, for example, AR glasses.

[0458] A "mobile information terminal" is a portable information processing device that has wireless communication capabilities, and includes, for example, a smartphone.

[0459] "Means of emotion analysis" refers to technology that identifies a speaker's emotions based on parameters such as tone and pitch of the voice.

[0460] "Visual display means" refers to a device or method for displaying text or emotional information on a screen or the like.

[0461] An embodiment of the present invention will be described. An example is given in which a system that displays voice communication as text in real time and analyzes emotions and displays them visually is used in a physical store. A user uses this system to facilitate smoother conversations with customers.

[0462] System Program

[0463] The server processes audio data using multiple pieces of hardware and software. Specifically, it has the following configuration:

[0464] Hardware: The servers are equipped with high-performance processors and large amounts of memory, which allows for the processing of large amounts of voice data in real time. Smart glasses and smartphones are used as terminals.

[0465] Software: The server runs voice recognition software (e.g., Whisper) and an emotion recognition engine. The smart glasses and smartphones have a dedicated application installed to send and receive voice data.

[0466] Processing flow

[0467] When a user wears the smart glasses and begins talking to a customer, their voice is captured by the glasses' microphone. This voice data is temporarily stored in the smart glasses, compressed, encrypted, and sent to a server. On the server side, the voice data is decoded and converted into text data using voice recognition software. At the same time, an emotion recognition engine analyzes emotions from the voice data and generates emotion data.

[0468] The generated text and emotion data are compressed and encrypted again and sent back to the device. The smart glasses decode this data and display it on the screen. This allows the user to not only visually confirm the content of the conversation with the customer in text, but also understand the customer's emotions.

[0469] Specific examples

[0470] For example, if a customer says, "Tell me more about this product," the microphone in the glasses captures this voice and sends it to the server. This voice is converted into text data, and the emotion recognition engine analyzes the emotion of "interest." The smart glasses' display then displays text information such as "Tell me more about this product (emotion: interest)." This allows the user (store clerk) to understand the customer's interest and provide appropriate explanations.

[0471] Prompt Sentence Examples

[0472] The following is an example of a prompt to be input to a generative AI model that transcribes user speech in real time and then recognizes and displays emotions:

[0473] "When a customer says, 'Tell me more about this product,' transcribe that speech in real time and recognize and display the emotion 'interested.'"

[0474] This will significantly improve customer service in physical stores and increase customer satisfaction.

[0475] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0476] Step 1:

[0477] The user puts on the smart glasses and begins talking to the customer, whose voice is captured in real time by the smart glasses' microphone.

[0478] How it works: The smart glasses' built-in microphone collects audio digitally. The input to this process is the customer's voice, and the output is digital audio data.

[0479] Step 2:

[0480] The device (smart glasses) temporarily stores the captured audio data and compresses and encrypts it.

[0481] How it works: Audio data is stored in temporary storage and encrypted using an encryption algorithm such as AES. The input to this process is digital audio data, and the output is compressed and encrypted audio data.

[0482] Step 3:

[0483] The device (smart glasses) sends compressed and encrypted audio data to the server using a secure protocol (e.g., HTTPS).

[0484] Specific operation: Using the data transmission library, compressed and encrypted audio data is uploaded to the server. The input of this process is the compressed and encrypted audio data, and the output is the data transmitted to the server.

[0485] Step 4:

[0486] The server decodes the received voice data and converts it into text data using voice recognition software (e.g., Whisper).

[0487] Specific operation: Decodes the audio data using a decoding library and calls the speech recognition API to generate text data. The input of this process is the received audio data, and the output is the text data.

[0488] Step 5:

[0489] The server uses an emotion recognition engine to analyze emotions from the voice data.

[0490] Specific operation: The emotion recognition engine uses analysis algorithms to extract emotional parameters (e.g., tone, pitch, etc.) from speech and generates emotional data. The input of this process is speech data, and the output is emotional data.

[0491] Step 6:

[0492] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the terminal.

[0493] Specific operation: Text data and emotion data are compressed and encrypted using the compression library and encryption library, and then sent to the terminal using the data transmission library. The input of this process is character data and emotion data, and the output is compressed and encrypted character data and emotion data.

[0494] Step 7:

[0495] The device (smart glasses) decodes the received text data and emotion data and displays it on the screen.

[0496] Specific operation: Decode using the decoding library, and display text and emotions on the AR display using the display library. The input of this process is the compressed and encrypted character data and emotion data, and the output is the text and emotion data displayed on the smart glasses display.

[0497] Step 8:

[0498] The user can check the content of the conversation and the emotions of the person they are talking to through the display on the smart glasses.

[0499] Specific operation: The user visually checks the text and emotional information displayed on the smart glasses display and responds to the customer. The input of this process is the text displayed on the display and emotional data, and the output is the user's behavior.

[0500] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0501] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0502] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0503] [Second embodiment]

[0504] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0505] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0506] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0507] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0508] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0509] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0510] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0511] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0512] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0513] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0514] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0515] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0516] This invention relates to a system for displaying voice communication as text in real time. The system converts voice into text and allows users to visually confirm the text when communicating with a conversation partner in person or over the phone.

[0517] In a face-to-face conversation

[0518] 1. The user puts on the AR glasses and talks to the other person.

[0519] The AR glasses have built-in microphones that capture audio as users speak.

[0520] 2. The device (AR glasses) captures audio in real time.

[0521] The voices of the person you are speaking to are collected as digital voice data.

[0522] 3. The device sends the collected voice data to the server.

[0523] The communication function of the AR glasses is used to send audio data to the server.

[0524] 4. The server analyzes the voice data and converts it into text data.

[0525] The server uses voice recognition technology such as Whisper to convert the audio into text.

[0526] 5. The server returns the converted character data to the terminal.

[0527] The text data is then sent back to the AR glasses.

[0528] 6. The device displays the received text data on the AR display.

[0529] Users can view the content of the conversation in real time as text.

[0530] Examples:

[0531] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the AR glasses, where "What time is the next meeting?" is displayed on the display. The user can visually confirm the content.

[0532] For telephone communication

[0533] 1. The user launches the calling app on their smartphone and makes a call.

[0534] The smartphone will begin recording the audio of both parties during the call.

[0535] 2. The device (smartphone) captures the audio from both parties.

[0536] The audio of the call is collected as digital data.

[0537] 3. The device sends the captured audio data to the server.

[0538] The smartphone encrypts the voice data it collects and sends it to a server.

[0539] 4. The server analyzes the voice data and converts it into text data.

[0540] The dialogue is converted into text using voice recognition technology such as the Whisper model.

[0541] 5. The server returns the converted character data to the terminal.

[0542] The text data is sent to the smartphone.

[0543] 6. The device displays the received text data on the smartphone screen.

[0544] Users can view the contents of phone conversations as text in real time.

[0545] Examples:

[0546] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server converts the audio into text and sends it back to the smartphone, where "Tell me about your new project" is displayed on the screen. The user can visually confirm the content.

[0547] This system converts speech into text in real time during face-to-face and remote communication, enabling smooth conversations even for people with hearing impairments or in noisy environments. This allows users to visually check the transcribed data, improving the quality of communication.

[0548] The processing flow will be explained below.

[0549] In a face-to-face conversation

[0550] Step 1:

[0551] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[0552] Step 2:

[0553] The user's voice is captured in real time by the microphone in the AR glasses.

[0554] Step 3:

[0555] The device (AR glasses) temporarily stores the captured audio data and compresses and encrypts it.

[0556] Step 4:

[0557] The device sends compressed and encrypted audio data to the server using a secure protocol.

[0558] Step 5:

[0559] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0560] Step 6:

[0561] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0562] Step 7:

[0563] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[0564] Step 8:

[0565] The device (AR glasses) decodes the received text data and displays it on the AR glasses' display.

[0566] Step 9:

[0567] Users can see what their conversation partner is saying in real time.

[0568] For telephone communication

[0569] Step 1:

[0570] The user launches the calling app on their smartphone and calls the person they want to call.

[0571] Step 2:

[0572] The user's voice and the voice of the other party are captured in real time by the smartphone's microphone.

[0573] Step 3:

[0574] The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[0575] Step 4:

[0576] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[0577] Step 5:

[0578] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0579] Step 6:

[0580] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0581] Step 7:

[0582] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[0583] Step 8:

[0584] The device (smartphone) decodes the received text data and displays it on the call app screen.

[0585] Step 9:

[0586] Users can view the contents of the call in real time as text.

[0587] This system allows face-to-face and telephone communications to be visually confirmed as text through a series of processes that captures voice in real time, converts it into text, and displays it.

[0588] Example 1

[0589] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0590] Conventional voice communication has the drawback of being difficult to communicate smoothly when the speaker is inaudible or when used in environments with a lot of background noise. Furthermore, people with hearing impairments and some elderly people have difficulty understanding spoken information, creating a need for a means to convert speech into text in real time. Furthermore, from the perspective of privacy protection, encryption technology is also needed to process voice data securely.

[0591] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0592] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into text data, means for returning the text data to the terminal, means for displaying the text data to the user, means for the means to use voice recognition technology to generate, transmit, and display visually identifiable text data in real time, and means for encrypting the voice data transmission and reception process. This allows users to visually confirm voice as text in real time, enabling smooth communication even in environments with a lot of background noise or for the hearing impaired. Furthermore, encryption of the voice data also enhances privacy protection.

[0593] The "means for accepting voice input" refers to a device or software function that captures the voice uttered by the user and environmental sounds and acquires the data necessary for subsequent processing.

[0594] "Means for capturing audio data" refers to the functionality of a device or software that digitally records audio input and converts it into a format suitable for further processing.

[0595] The "means for transmitting audio data to a server" refers to a device or software function that transfers captured audio data to a server via a communication line.

[0596] "Means for converting voice data into text data" refers to the functionality of a device or software within the server that converts voice data into text using voice recognition technology.

[0597] "Means for returning character data to the terminal" refers to the function of a device or software that sends the text data converted by the server back to the terminal via a communication line.

[0598] The "means for displaying character data to the user" refers to a device or software function that visually presents the returned text data to the user on a terminal display or the like.

[0599] "Means of using voice recognition technology" refers to technology or software functions for analyzing voice data and converting it into text with high accuracy.

[0600] "Encryption means" refers to the function of a device or software that encrypts data to maintain confidentiality during the communication of voice data and text data.

[0601] "Augmented reality glasses" are glasses-type devices that can be worn by the user to overlay virtual information (such as text data) onto the real world.

[0602] A "mobile phone" is a small portable communication terminal capable of voice calls and data communications.

[0603] This invention relates to a system that displays voice communication as text in real time, and is a system that allows a user to convert voice into text and visually confirm it when communicating with a conversation partner in person or over the phone.

[0604] This system is implemented as follows: First, a user uses either augmented reality glasses or a mobile phone. When a user wants to have a face-to-face conversation, the user puts on the augmented reality glasses and starts the conversation. The augmented reality glasses have a built-in microphone that captures the voice of the person they are talking to in real time. When a user wants to have a conversation over the phone, the voice data is captured using the mobile phone's calling app.

[0605] The device (augmented reality glasses or mobile phone) encodes the captured voice data and transmits it over a communication line to a server, where it is received and converted into text using voice recognition technology. Specifically, a voice recognition model such as OpenAI's Whisper is used.

[0606] Once the voice has been converted into text data, the server re-encrypts the text data and sends it back to the terminal via the communication line. The terminal then decrypts the received text data and displays it on a display so that the user can visually confirm it. This process occurs in real time, so the user can immediately see the contents of the conversation in text form.

[0607] The system also uses encryption technology to protect the privacy of users' voice data and converted text data, allowing users to use the system with peace of mind.

[0608] Specific examples

[0609] In a face-to-face conversation: When a user says, "What time is the next meeting?", the augmented reality glasses capture the voice and send it to the server. The server converts the voice into text data and sends it back to the augmented reality glasses. The user can see the text "What time is the next meeting?" displayed on the display inside the AR glasses.

[0610] In the case of telephone communication: When a user says "Tell me about your new project" on the phone, the mobile phone captures this voice and sends it to the server. The server converts the voice into text data and sends it back to the mobile phone. The user can see the text "Tell me about your new project" on the mobile phone screen.

[0611] Prompt Sentence Examples

[0612] 1. "What time is the next meeting?"

[0613] 2. "Tell me about your new project."

[0614] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0615] Step 1:

[0616] The device (augmented reality glasses or mobile phone) accepts voice input.

[0617] Input: User's voice.

[0618] How it works: In the case of augmented reality glasses, a microphone picks up what the user says and stores it as digital audio data. In the case of a mobile phone, the voice is recorded through a calling app.

[0619] Output: Captured as digital audio data and temporarily stored in memory.

[0620] Step 2:

[0621] The device sends the captured audio data to the server.

[0622] Input: Digital audio data.

[0623] What happens: Your device uses your internet connection to encrypt and send audio data to a server. Wi-Fi or cellular data is used.

[0624] Output: The encrypted audio data arrives at the server.

[0625] Step 3:

[0626] The server receives the audio data and begins analyzing it.

[0627] Input: Encrypted audio data.

[0628] Specific operation: The server decodes the received voice data and converts it into text using voice recognition technology (e.g., OpenAI's Whisper).

[0629] Output: The parsed text data is generated.

[0630] Step 4:

[0631] The server returns the converted character data to the terminal.

[0632] Input: Parsed text data.

[0633] Specific operation: The server encrypts the text data and sends it to the device over the communication line, again using Wi-Fi or mobile data.

[0634] Output: Encrypted text data arrives at the terminal.

[0635] Step 5:

[0636] The terminal decodes the received character data and displays it to the user.

[0637] Input: Encrypted text data.

[0638] Specific operation: The device decodes the received text data and displays it on the display in the case of augmented reality glasses, or on the screen in the case of a mobile phone.

[0639] Output: Text data that can be visually confirmed by the user.

[0640] Specific examples

[0641] Prompt Sentence Examples

[0642] 1. "What time is the next meeting?"

[0643] 2. "Tell me about your new project."

[0644] (Application example 1)

[0645] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0646] Currently, it is difficult for hearing-impaired employees to communicate with customers in noisy brick-and-mortar stores. It is also difficult for employees to clearly hear what customers are saying when they are wearing masks or in crowded environments. In these environments, an effective method for smoothly conducting conversations with customers is needed.

[0647] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0648] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into character data, means for returning the character data to the terminal, means for displaying the character data on a visual information display device, and means for the visual information display device to be used by employees of the physical store to communicate with customers. This allows employees of the physical store to visually confirm what customers are saying, enabling smoother communication.

[0649] A "means for accepting voice input" is a device or part of a system that collects voice data, recognizes voice through hardware such as a microphone, and processes it as digital data.

[0650] "Means for capturing voice input as audio data" refers to a device or process that records received audio in a digital data format and converts it into analyzable data.

[0651] "Means for transmitting audio data to a server" refers to a communication interface or protocol for transferring captured audio data to a remote server via a network.

[0652] "Means for converting voice data into text data" refers to the speech recognition technology or software used to analyze received voice data and generate a corresponding string of characters.

[0653] "Means for returning character data to the terminal" refers to the communication functions and protocols for returning character data generated by the server to the original terminal via the network.

[0654] "Means for displaying character data on a visual information display device" refers to a display device such as a display or AR glasses, and its control system, for visually presenting the character data returned to the terminal to the user.

[0655] A "visual information display device" is an electronic device that provides information visually to a user, and includes displays, AR glasses, head-mounted displays, etc.

[0656] A "physical store" is a physical space, a place where products and services are offered and where store staff and customers can communicate face-to-face.

[0657] A "personal digital assistant" is a portable electronic device capable of communicating voice and data, including smartphones and tablets.

[0658] This invention provides a system that converts voice input into text and displays it in real time when employees of a brick-and-mortar store communicate with customers. This system accepts voice input, converts it into text data, and displays it to employees on a visual information display device, enabling appropriate communication even in noisy environments or for those with hearing impairments.

[0659] Hardware and software used

[0660] This system requires the following hardware and software:

[0661] Hardware:

[0662] A microphone that accepts voice input (built into the visual information display device)

[0663] Visual information display devices (e.g., AR glasses, smart glasses)

[0664] Mobile information terminals (e.g. smartphones)

[0665] software:

[0666] Audio Capture Software

[0667] Communication Interface

[0668] Speech recognition engine (e.g. Whisper model)

[0669] Data Display Software

[0670] The server converts the voice data into text data using voice recognition technology, and the terminal receives the text data and displays it on a visual information display device.

[0671] Data processing and calculation methods

[0672] 1. Audio capture:

[0673] The user accepts voice input through the visual information display device. For example, a store employee wears smart glasses and converses with a customer.

[0674] 2. Sending audio data:

[0675] The device then sends the captured audio data to the server, where it encrypts the data using its communication function to ensure secure transmission.

[0676] 3. Speech recognition and conversion to text data:

[0677] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into corresponding text data.

[0678] 4. Returning character data:

[0679] The server returns the character data to the terminal, which displays the character data on a visual information display device.

[0680] 5. Character display:

[0681] Users can check text data in real time through the visual information display device, which allows them to visually recognize what the customer is saying and respond appropriately.

[0682] Specific examples

[0683] For example, if a customer says to a store employee, "Please explain this product to me," the smart glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the smart glasses as "Please explain this product to me." As a result, the store employee can see the text displayed on the visual information display device and understand the content of the conversation.

[0684] Example prompt for a generative AI model:

[0685] "Please tell me the specific steps to implement a system that transcribes conversations in a physical store in real time. Please provide a detailed explanation including voice capture, sending the voice data to the server, voice recognition, and displaying the text data."

[0686] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0687] Step 1:

[0688] A user wears a visual information display device (e.g., smart glasses) and talks with a customer.

[0689] Input: Customer voice

[0690] How it works: A microphone built into the visual display captures sound.

[0691] Output: Digital audio data

[0692] Step 2:

[0693] The terminal transmits the captured audio data to a server via a network.

[0694] Input: Digital audio data

[0695] How it works: The device's communications interface encrypts the audio data and sends it to the server.

[0696] Output: Audio data sent to the server

[0697] Step 3:

[0698] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into text data.

[0699] Input: Audio data transferred to the server

[0700] How it works: The server's speech recognition engine analyzes the audio data and generates corresponding text data.

[0701] Output: Character data

[0702] Step 4:

[0703] The server returns the generated character data to the terminal.

[0704] Input: Character data

[0705] Operation: The server's communication interface sends character data to the terminal.

[0706] Output: Character data sent to the terminal

[0707] Step 5:

[0708] The terminal displays the received character data on a visual information display device in real time.

[0709] Input: Character data sent to the terminal

[0710] Operation: The visual information display device displays the received character data on the display.

[0711] Output: Text data that can be visually confirmed by the user

[0712] Specific examples

[0713] Let's consider what happens when a customer asks an employee in a physical store, "Please explain this product to me."

[0714] Step 1: The microphone built into the visual display captures the audio, "Please describe this product."

[0715] Step 2: The device's communication interface encrypts this voice data and sends it to the server.

[0716] Step 3: The server's speech recognition engine (e.g., Whisper model) analyzes the voice data and generates text data such as "Please describe this product."

[0717] Step 4: The server's communication interface sends the character data to the terminal.

[0718] Step 5: The visual information display device displays the received text data on the screen, allowing the employee to confirm the content as text.

[0719] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0720] This invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts voice into text and analyzes and visually confirms the user's emotions.

[0721] In a face-to-face conversation

[0722] 1. The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[0723] 2. The device (AR glasses) captures the voice in real time. The voice of the person you are talking to is collected as digital audio data.

[0724] 3. The device temporarily stores the collected voice data and compresses and encrypts it.

[0725] 4. The device sends the compressed and encrypted audio data to the server. This communication uses a secure protocol.

[0726] 5. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0727] 6. The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0728] 7. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[0729] 8. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the terminal.

[0730] 9. The device (AR glasses) decodes the received text data and emotion data and displays it on the AR glasses display.

[0731] 10. Users can check the content of the conversation as well as the emotional state of the person they are talking to in real time.

[0732] Examples:

[0733] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server transcribes the voice, and at the same time, the emotion recognition engine analyzes whether the user is speaking in a serious tone, and the message "What time is the next meeting? (Emotion: Serious)" is displayed on the AR glasses' display. The user can visually confirm the content of the conversation and the emotion.

[0734] For telephone communication

[0735] 1. A user launches a calling app on their smartphone and makes a call to the other party. The user's voice and the other party's voice are captured in real time by the smartphone's microphone.

[0736] 2. The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[0737] 3. The device sends the compressed and encrypted audio data to the server, again using a secure protocol.

[0738] 4. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[0739] 5. The server analyzes the voice data and converts it into text data, which is then stored in internal memory.

[0740] 6. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[0741] 7. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the device.

[0742] 8. The device (smartphone) decodes the received text data and emotion data and displays it on the call app screen.

[0743] 9. Users can check the content of the call as well as the emotional state of the other party in real time.

[0744] Examples:

[0745] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server transcribes the audio, and at the same time, an emotion recognition engine analyzes whether the user is speaking in an interested tone, and the message "Tell me about your new project (emotion: interest)" is displayed on the smartphone screen. The user can visually confirm the content of the call and the emotion.

[0746] This system improves the quality of face-to-face and telephone communication by capturing speech in real time, transcribing it, analyzing emotions, and displaying the results simultaneously. This process allows users to visually confirm both the content and emotions of the communication, enabling smoother dialogue.

[0747] The processing flow will be explained below.

[0748] In a face-to-face conversation

[0749] Step 1:

[0750] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[0751] Step 2:

[0752] The device (AR glasses) captures audio in real time, and the voice of the person you are talking to is collected as digital audio data.

[0753] Step 3:

[0754] The device temporarily stores the collected voice data in its internal memory and simultaneously compresses and encrypts it.

[0755] Step 4:

[0756] The device sends compressed and encrypted audio data to the server using a secure protocol.

[0757] Step 5:

[0758] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[0759] Step 6:

[0760] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0761] Step 7:

[0762] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[0763] Step 8:

[0764] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[0765] Step 9:

[0766] The device decodes the received text data and emotion data and displays it on the AR glasses' display.

[0767] Step 10:

[0768] Users can check the content of the conversation as well as the emotions of their conversation partner in real time.

[0769] For telephone communication

[0770] Step 1:

[0771] A user starts a calling app on their smartphone and makes a call to the other party. The voices of both parties are captured in real time by the smartphone's microphone during the call.

[0772] Step 2:

[0773] The device (smartphone) temporarily stores the captured audio data in its internal memory, and simultaneously compresses and encrypts it.

[0774] Step 3:

[0775] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[0776] Step 4:

[0777] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[0778] Step 5:

[0779] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[0780] Step 6:

[0781] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[0782] Step 7:

[0783] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[0784] Step 8:

[0785] The device decodes the received text data and emotion data and displays them on the call app screen.

[0786] Step 9:

[0787] Users can see the content of the call as well as the emotions of the person they are talking to in real time.

[0788] In this way, a system is realized that converts voice communication into text and then analyzes and displays emotional data, improving the quality of face-to-face and telephone communication.

[0789] Example 2

[0790] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0791] In modern voice communication, understanding the emotions of the person you're speaking to is extremely important. However, conventional systems only transcribe speech and are unable to analyze emotions, making it difficult for users to fully understand the content of the conversation. Furthermore, there was a lack of technology that could consistently perform real-time transcription and emotion analysis, preventing efficient communication support. This could result in a decline in the quality of the conversation and the risk of misunderstandings and breakdowns in communication.

[0792] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0793] In this invention, the server includes means for decoding voice data and converting it into text data, means for analyzing emotions from the voice data, and means for compressing and encrypting the text data and emotional data, thereby enabling real-time transcription and emotional analysis of voice communication.

[0794] "Voice input" refers to words or dialogue spoken by a user.

[0795] "Audio Data" means captured audio input stored in digital form.

[0796] "Compression" is the technique of reducing the size of data so that it can be stored and transmitted efficiently.

[0797] "Encryption" is the process of converting data into an unintelligible form for protection.

[0798] A "server" is a computer system responsible for analyzing and processing audio data.

[0799] "Decoding" is the technique of restoring encrypted data to its original form.

[0800] "Character data" is data in text format into which voice data has been converted.

[0801] "Emotion analysis" is a technology that identifies a speaker's emotions by analyzing the tone and pitch of a voice.

[0802] A "terminal" is a device that captures and displays audio data.

[0803] "Display" refers to the act of visually showing text and emotion data to the user.

[0804] A "head-mounted display" is a display device that is worn on the head.

[0805] A "personal digital assistant" is a portable device that captures and displays audio data.

[0806] The present invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts the voice into text and analyzes and visually confirms the user's emotions.

[0807] This system consists of a terminal that accepts voice input and captures the voice data, and a server that analyzes the voice data. Specifically, the following hardware and software are used:

[0808] Hardware

[0809] 1. Terminal: A device that captures and displays audio data. For face-to-face conversations, a head-mounted display is used, and for telephone communication, a personal digital assistant is used.

[0810] 2. Server: A computer system that decodes audio data, converts it to text data, and analyzes emotions.

[0811] software

[0812] 1. Speech recognition software: converts voice data into text data. For example, we will use Whisper.

[0813] 2. Emotion recognition engine: Identifies emotions by analyzing voice tone, pitch, etc.

[0814] Processing flow

[0815] In a face-to-face conversation

[0816] 1. The user puts on the head-mounted display and prepares to start a conversation with the other person. The display's microphone prepares for voice capture.

[0817] 2. The device (head-mounted display) captures audio in real time and temporarily stores it as digital audio data.

[0818] 3. The device compresses and then encrypts the collected audio data.

[0819] 4. The device sends the audio data to the server using a secure communication protocol such as TLS or HTTPS.

[0820] 5. The server decodes the received voice data and inputs it into the voice recognition software.

[0821] 6. The server analyzes the voice data and converts it into text data.

[0822] 7. The server uses an emotion recognition engine to analyze emotions from the voice data.

[0823] 8. The server re-compresses and encrypts the converted text data and emotion data and sends them back to the terminal.

[0824] 9. The device (head-mounted display) decodes the voice and emotion data and displays it on the display.

[0825] 10. The user visually confirms the displayed text data and emotional information.

[0826] For telephone communication

[0827] 1. The user launches the calling app on their mobile device and calls the person they want to call. The microphone on the mobile device prepares to capture audio.

[0828] 2. The device (mobile information terminal) captures the audio data in real time and temporarily stores it.

[0829] 3. The device compresses and encrypts the captured audio data.

[0830] 4. The device sends the audio data to the server using a secure communication protocol.

[0831] 5. The server decodes the received data and inputs it into the voice recognition software.

[0832] 6. The server analyzes the voice data and converts it into text data.

[0833] 7. The server analyzes the voice data using an emotion recognition engine to identify the emotion.

[0834] 8. The server re-compresses the converted text data and emotion data, encrypts them, and returns them to the terminal.

[0835] 9. The device (mobile information terminal) decodes the data and displays it in the calling application.

[0836] 10. The user visually confirms the text data and emotional information.

[0837] Specific examples

[0838] As an example of a prompt sentence, if a user says "What time is the next meeting?", the AR glasses will capture the voice and send it to the server. The server will convert the voice into text and analyze it with an emotion recognition engine, which will then display "What time is the next meeting? (Emotion: Serious)" on the AR glasses. The user can see this display in real time.

[0839] Furthermore, in the case of telephone communication, when a user says, "Tell me about your new project," the mobile information terminal captures the voice and sends it to the server. The server converts it into text data, which is displayed on the mobile information terminal as, "Tell me about your new project (emotion: interest)."

[0840] This system can improve the quality of face-to-face and telephone communication by transcribing speech in real time, analyzing emotions, and displaying the results.

[0841] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0842] In a face-to-face conversation

[0843] Step 1:

[0844] The user puts on the head-mounted display and prepares to start a conversation with the other person. The microphone of the head-mounted display prepares to capture voice.

[0845] Input: The user puts on the head-mounted display and begins interacting.

[0846] Output: The microphone is now operational and ready to capture audio.

[0847] Step 2:

[0848] The device (head-mounted display) captures surrounding sounds in real time, and the microphone collects the voice of the person you are talking to with high accuracy and temporarily stores it as digital audio data.

[0849] Input: Voice of the person you are talking to.

[0850] Output: Temporarily stored as digital audio data.

[0851] Step 3:

[0852] The terminal compresses the collected audio data and then encrypts it.

[0853] Input: Digital audio data.

[0854] Output: Compressed and encrypted audio data.

[0855] Step 4:

[0856] The device sends compressed and encrypted audio data to the server using secure communication protocols such as TLS and HTTPS.

[0857] Input: Compressed and encrypted audio data.

[0858] Output: Sent to the server.

[0859] Step 5:

[0860] The server decodes the received audio data and restores it to the original audio data.

[0861] Input: Encrypted audio data.

[0862] Output: Decoded audio data.

[0863] Step 6:

[0864] The server inputs the recovered voice data into voice recognition software (e.g., Whisper) and converts it into text data, which is then temporarily stored in internal memory.

[0865] Input: Decoded audio data.

[0866] Output: Character data.

[0867] Step 7:

[0868] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes the tone, pitch and other parameters of the voice to identify emotions.

[0869] Input: Decoded audio data.

[0870] Output: Emotion data.

[0871] Step 8:

[0872] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[0873] Input: Text data and emotion data.

[0874] Output: Compressed and encrypted text and emotion data.

[0875] Step 9:

[0876] The terminal (head-mounted display) decodes the voice and emotion data and displays it on the display.

[0877] Input: Compressed and encrypted character data and emotion data.

[0878] Output: Decoded character data and emotion data are displayed on the screen.

[0879] Step 10:

[0880] The user visually checks the displayed character data and emotion information.

[0881] Input: Character data and emotion data displayed on the display.

[0882] Output: User confirmation.

[0883] For telephone communication

[0884] Step 1:

[0885] The user launches a calling application on the mobile information terminal and makes a call to the other party. The microphone on the mobile information terminal prepares for voice capture.

[0886] Input: The user launches a calling app on a mobile information terminal and starts a call.

[0887] Output: The microphone is now operational and ready to capture audio.

[0888] Step 2:

[0889] The device (mobile information terminal) captures the voices of the user and the other party in real time, and temporarily stores them as digital voice data.

[0890] Input: Caller and user voice.

[0891] Output: Temporarily stored as digital audio data.

[0892] Step 3:

[0893] The device compresses the captured audio data and then encrypts it.

[0894] Input: Digital audio data.

[0895] Output: Compressed and encrypted audio data.

[0896] Step 4:

[0897] The device sends compressed and encrypted audio data to the server using a secure communication protocol.

[0898] Input: Compressed and encrypted audio data.

[0899] Output: Sent to the server.

[0900] Step 5:

[0901] The server decodes the received audio data and restores it to the original audio data.

[0902] Input: Encrypted audio data.

[0903] Output: Decoded audio data.

[0904] Step 6:

[0905] Similarly, the server inputs the voice data into voice recognition software and converts it into text data, which is then temporarily stored in internal memory.

[0906] Input: Decoded audio data.

[0907] Output: Character data.

[0908] Step 7:

[0909] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes parameters such as tone, pitch, speed, and volume of the voice to identify emotions.

[0910] Input: Decoded audio data.

[0911] Output: Emotion data.

[0912] Step 8:

[0913] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[0914] Input: Text data and emotion data.

[0915] Output: Compressed and encrypted text and emotion data.

[0916] Step 9:

[0917] The terminal (mobile information terminal) decodes the received text data and emotion data and displays them on the screen of the calling app.

[0918] Input: Compressed and encrypted character data and emotion data.

[0919] Output: Decoded character data and emotion data are displayed on the screen.

[0920] Step 10:

[0921] The user visually checks the displayed character data and emotion information.

[0922] Input: Character data and emotion data displayed on the screen.

[0923] Output: User confirmation.

[0924] (Application example 2)

[0925] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0926] In today's brick-and-mortar stores, it is difficult to accurately grasp a customer's intentions and emotions when dealing with customers, making improving customer satisfaction a challenge. If it were possible to instantly understand a customer's emotions on-site, more appropriate responses could be made. Furthermore, displaying voice communication as text in real time would clearly understand the customer's intentions and reduce misunderstandings. However, existing systems that achieve these goals are limited, and in many cases, problems include insufficient emotion analysis capabilities and insufficient system response speed.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0928] In this invention, the server includes means for receiving voice input in real time, means for converting voice data into text data using voice recognition software, means for analyzing emotions from the voice data, and means for returning the analyzed emotional data and text data to the terminal. This makes it possible to convert conversations with customers into text in real time and to visually display the customer's emotions. This allows store clerks to instantly understand the customer's emotions and take appropriate action, contributing to improved customer satisfaction.

[0929] "Voice input" is the means by which human speech is received in digital form.

[0930] "Audio data" is data that represents audio as a digital signal.

[0931] A "server" is a computer system that processes information and manages data on a network.

[0932] "Character data" refers to text data converted from voice data using voice recognition technology.

[0933] "Terminal" means a device with input and output capabilities that can be directly operated by a user.

[0934] A "user" is a person who uses the system.

[0935] "Visual device" refers to a device that has the function of displaying visual information, and includes, for example, AR glasses.

[0936] A "mobile information terminal" is a portable information processing device that has wireless communication capabilities, and includes, for example, a smartphone.

[0937] "Means of emotion analysis" refers to technology that identifies a speaker's emotions based on parameters such as tone and pitch of the voice.

[0938] "Visual display means" refers to a device or method for displaying text or emotional information on a screen or the like.

[0939] An embodiment of the present invention will be described. An example is given in which a system that displays voice communication as text in real time and analyzes emotions and displays them visually is used in a physical store. A user uses this system to facilitate smoother conversations with customers.

[0940] System Program

[0941] The server processes audio data using multiple pieces of hardware and software. Specifically, it has the following configuration:

[0942] Hardware: The servers are equipped with high-performance processors and large amounts of memory, which allows for the processing of large amounts of voice data in real time. Smart glasses and smartphones are used as terminals.

[0943] Software: The server runs voice recognition software (e.g., Whisper) and an emotion recognition engine. The smart glasses and smartphones have a dedicated application installed to send and receive voice data.

[0944] Processing flow

[0945] When a user wears the smart glasses and begins talking to a customer, their voice is captured by the glasses' microphone. This voice data is temporarily stored in the smart glasses, compressed, encrypted, and sent to a server. On the server side, the voice data is decoded and converted into text data using voice recognition software. At the same time, an emotion recognition engine analyzes emotions from the voice data and generates emotion data.

[0946] The generated text and emotion data are compressed and encrypted again and sent back to the device. The smart glasses decode this data and display it on the screen. This allows the user to not only visually confirm the content of the conversation with the customer in text, but also understand the customer's emotions.

[0947] Specific examples

[0948] For example, if a customer says, "Tell me more about this product," the microphone in the glasses captures this voice and sends it to the server. This voice is converted into text data, and the emotion recognition engine analyzes the emotion of "interest." The smart glasses' display then displays text information such as "Tell me more about this product (emotion: interest)." This allows the user (store clerk) to understand the customer's interest and provide appropriate explanations.

[0949] Prompt Sentence Examples

[0950] The following is an example of a prompt to be input to a generative AI model that transcribes user speech in real time and then recognizes and displays emotions:

[0951] "When a customer says, 'Tell me more about this product,' transcribe that speech in real time and recognize and display the emotion 'interested.'"

[0952] This will significantly improve customer service in physical stores and increase customer satisfaction.

[0953] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0954] Step 1:

[0955] The user puts on the smart glasses and begins talking to the customer, whose voice is captured in real time by the smart glasses' microphone.

[0956] How it works: The smart glasses' built-in microphone collects audio digitally. The input to this process is the customer's voice, and the output is digital audio data.

[0957] Step 2:

[0958] The device (smart glasses) temporarily stores the captured audio data and compresses and encrypts it.

[0959] How it works: Audio data is stored in temporary storage and encrypted using an encryption algorithm such as AES. The input to this process is digital audio data, and the output is compressed and encrypted audio data.

[0960] Step 3:

[0961] The device (smart glasses) sends compressed and encrypted audio data to the server using a secure protocol (e.g., HTTPS).

[0962] Specific operation: Using the data transmission library, compressed and encrypted audio data is uploaded to the server. The input of this process is the compressed and encrypted audio data, and the output is the data transmitted to the server.

[0963] Step 4:

[0964] The server decodes the received voice data and converts it into text data using voice recognition software (e.g., Whisper).

[0965] Specific operation: Decodes the audio data using a decoding library and calls the speech recognition API to generate text data. The input of this process is the received audio data, and the output is the text data.

[0966] Step 5:

[0967] The server uses an emotion recognition engine to analyze emotions from the voice data.

[0968] Specific operation: The emotion recognition engine uses analysis algorithms to extract emotional parameters (e.g., tone, pitch, etc.) from speech and generates emotional data. The input of this process is speech data, and the output is emotional data.

[0969] Step 6:

[0970] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the terminal.

[0971] Specific operation: Text data and emotion data are compressed and encrypted using the compression library and encryption library, and then sent to the terminal using the data transmission library. The input of this process is character data and emotion data, and the output is compressed and encrypted character data and emotion data.

[0972] Step 7:

[0973] The device (smart glasses) decodes the received text data and emotion data and displays it on the screen.

[0974] Specific operation: Decode using the decoding library, and display text and emotions on the AR display using the display library. The input of this process is the compressed and encrypted character data and emotion data, and the output is the text and emotion data displayed on the smart glasses display.

[0975] Step 8:

[0976] The user can check the content of the conversation and the emotions of the person they are talking to through the display on the smart glasses.

[0977] Specific operation: The user visually checks the text and emotional information displayed on the smart glasses display and responds to the customer. The input of this process is the text displayed on the display and emotional data, and the output is the user's behavior.

[0978] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0979] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0980] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0981] [Third embodiment]

[0982] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0983] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0984] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0985] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0986] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0988] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0989] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0990] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0993] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0994] This invention relates to a system for displaying voice communication as text in real time. The system converts voice into text and allows users to visually confirm the text when communicating with a conversation partner in person or over the phone.

[0995] In a face-to-face conversation

[0996] 1. The user puts on the AR glasses and talks to the other person.

[0997] The AR glasses have built-in microphones that capture audio as users speak.

[0998] 2. The device (AR glasses) captures audio in real time.

[0999] The voices of the person you are speaking to are collected as digital voice data.

[1000] 3. The device sends the collected voice data to the server.

[1001] The communication function of the AR glasses is used to send audio data to the server.

[1002] 4. The server analyzes the voice data and converts it into text data.

[1003] The server uses voice recognition technology such as Whisper to convert the audio into text.

[1004] 5. The server returns the converted character data to the terminal.

[1005] The text data is then sent back to the AR glasses.

[1006] 6. The device displays the received text data on the AR display.

[1007] Users can view the content of the conversation in real time as text.

[1008] Examples:

[1009] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the AR glasses, where "What time is the next meeting?" is displayed on the display. The user can visually confirm the content.

[1010] For telephone communication

[1011] 1. The user launches the calling app on their smartphone and makes a call.

[1012] The smartphone will begin recording the audio of both parties during the call.

[1013] 2. The device (smartphone) captures the audio from both parties.

[1014] The audio of the call is collected as digital data.

[1015] 3. The device sends the captured audio data to the server.

[1016] The smartphone encrypts the voice data it collects and sends it to a server.

[1017] 4. The server analyzes the voice data and converts it into text data.

[1018] The dialogue is converted into text using voice recognition technology such as the Whisper model.

[1019] 5. The server returns the converted character data to the terminal.

[1020] The text data is sent to the smartphone.

[1021] 6. The device displays the received text data on the smartphone screen.

[1022] Users can view the contents of phone conversations as text in real time.

[1023] Examples:

[1024] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server converts the audio into text and sends it back to the smartphone, where "Tell me about your new project" is displayed on the screen. The user can visually confirm the content.

[1025] This system converts speech into text in real time during face-to-face and remote communication, enabling smooth conversations even for people with hearing impairments or in noisy environments. This allows users to visually check the transcribed data, improving the quality of communication.

[1026] The processing flow will be explained below.

[1027] In a face-to-face conversation

[1028] Step 1:

[1029] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[1030] Step 2:

[1031] The user's voice is captured in real time by the microphone in the AR glasses.

[1032] Step 3:

[1033] The device (AR glasses) temporarily stores the captured audio data and compresses and encrypts it.

[1034] Step 4:

[1035] The device sends compressed and encrypted audio data to the server using a secure protocol.

[1036] Step 5:

[1037] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1038] Step 6:

[1039] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1040] Step 7:

[1041] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[1042] Step 8:

[1043] The device (AR glasses) decodes the received text data and displays it on the AR glasses' display.

[1044] Step 9:

[1045] Users can see what their conversation partner is saying in real time.

[1046] For telephone communication

[1047] Step 1:

[1048] The user launches the calling app on their smartphone and calls the person they want to call.

[1049] Step 2:

[1050] The user's voice and the voice of the other party are captured in real time by the smartphone's microphone.

[1051] Step 3:

[1052] The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[1053] Step 4:

[1054] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[1055] Step 5:

[1056] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1057] Step 6:

[1058] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1059] Step 7:

[1060] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[1061] Step 8:

[1062] The device (smartphone) decodes the received text data and displays it on the call app screen.

[1063] Step 9:

[1064] Users can view the contents of the call in real time as text.

[1065] This system allows face-to-face and telephone communications to be visually confirmed as text through a series of processes that captures voice in real time, converts it into text, and displays it.

[1066] Example 1

[1067] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1068] Conventional voice communication has the drawback of being difficult to communicate smoothly when the speaker is inaudible or when used in environments with a lot of background noise. Furthermore, people with hearing impairments and some elderly people have difficulty understanding spoken information, creating a need for a means to convert speech into text in real time. Furthermore, from the perspective of privacy protection, encryption technology is also needed to process voice data securely.

[1069] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1070] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into text data, means for returning the text data to the terminal, means for displaying the text data to the user, means for the means to use voice recognition technology to generate, transmit, and display visually identifiable text data in real time, and means for encrypting the voice data transmission and reception process. This allows users to visually confirm voice as text in real time, enabling smooth communication even in environments with a lot of background noise or for the hearing impaired. Furthermore, encryption of the voice data also enhances privacy protection.

[1071] The "means for accepting voice input" refers to a device or software function that captures the voice uttered by the user and environmental sounds and acquires the data necessary for subsequent processing.

[1072] "Means for capturing audio data" refers to the functionality of a device or software that digitally records audio input and converts it into a format suitable for further processing.

[1073] The "means for transmitting audio data to a server" refers to a device or software function that transfers captured audio data to a server via a communication line.

[1074] "Means for converting voice data into text data" refers to the functionality of a device or software within the server that converts voice data into text using voice recognition technology.

[1075] "Means for returning character data to the terminal" refers to the function of a device or software that sends the text data converted by the server back to the terminal via a communication line.

[1076] The "means for displaying character data to the user" refers to a device or software function that visually presents the returned text data to the user on a terminal display or the like.

[1077] "Means of using voice recognition technology" refers to technology or software functions for analyzing voice data and converting it into text with high accuracy.

[1078] "Encryption means" refers to the function of a device or software that encrypts data to maintain confidentiality during the communication of voice data and text data.

[1079] "Augmented reality glasses" are glasses-type devices that can be worn by the user to overlay virtual information (such as text data) onto the real world.

[1080] A "mobile phone" is a small portable communication terminal capable of voice calls and data communications.

[1081] This invention relates to a system that displays voice communication as text in real time, and is a system that allows a user to convert voice into text and visually confirm it when communicating with a conversation partner in person or over the phone.

[1082] This system is implemented as follows: First, a user uses either augmented reality glasses or a mobile phone. When a user wants to have a face-to-face conversation, the user puts on the augmented reality glasses and starts the conversation. The augmented reality glasses have a built-in microphone that captures the voice of the person they are talking to in real time. When a user wants to have a conversation over the phone, the voice data is captured using the mobile phone's calling app.

[1083] The device (augmented reality glasses or mobile phone) encodes the captured voice data and transmits it over a communication line to a server, where it is received and converted into text using voice recognition technology. Specifically, a voice recognition model such as OpenAI's Whisper is used.

[1084] Once the voice has been converted into text data, the server re-encrypts the text data and sends it back to the terminal via the communication line. The terminal then decrypts the received text data and displays it on a display so that the user can visually confirm it. This process occurs in real time, so the user can immediately see the contents of the conversation in text form.

[1085] The system also uses encryption technology to protect the privacy of users' voice data and converted text data, allowing users to use the system with peace of mind.

[1086] Specific examples

[1087] In a face-to-face conversation: When a user says, "What time is the next meeting?", the augmented reality glasses capture the voice and send it to the server. The server converts the voice into text data and sends it back to the augmented reality glasses. The user can see the text "What time is the next meeting?" displayed on the display inside the AR glasses.

[1088] In the case of telephone communication: When a user says "Tell me about your new project" on the phone, the mobile phone captures this voice and sends it to the server. The server converts the voice into text data and sends it back to the mobile phone. The user can see the text "Tell me about your new project" on the mobile phone screen.

[1089] Prompt Sentence Examples

[1090] 1. "What time is the next meeting?"

[1091] 2. "Tell me about your new project."

[1092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1093] Step 1:

[1094] The device (augmented reality glasses or mobile phone) accepts voice input.

[1095] Input: User's voice.

[1096] How it works: In the case of augmented reality glasses, a microphone picks up what the user says and stores it as digital audio data. In the case of a mobile phone, the voice is recorded through a calling app.

[1097] Output: Captured as digital audio data and temporarily stored in memory.

[1098] Step 2:

[1099] The device sends the captured audio data to the server.

[1100] Input: Digital audio data.

[1101] What happens: Your device uses your internet connection to encrypt and send audio data to a server. Wi-Fi or cellular data is used.

[1102] Output: The encrypted audio data arrives at the server.

[1103] Step 3:

[1104] The server receives the audio data and begins analyzing it.

[1105] Input: Encrypted audio data.

[1106] Specific operation: The server decodes the received voice data and converts it into text using voice recognition technology (e.g., OpenAI's Whisper).

[1107] Output: The parsed text data is generated.

[1108] Step 4:

[1109] The server returns the converted character data to the terminal.

[1110] Input: Parsed text data.

[1111] Specific operation: The server encrypts the text data and sends it to the device over the communication line, again using Wi-Fi or mobile data.

[1112] Output: Encrypted text data arrives at the terminal.

[1113] Step 5:

[1114] The terminal decodes the received character data and displays it to the user.

[1115] Input: Encrypted text data.

[1116] Specific operation: The device decodes the received text data and displays it on the display in the case of augmented reality glasses, or on the screen in the case of a mobile phone.

[1117] Output: Text data that can be visually confirmed by the user.

[1118] Specific examples

[1119] Prompt Sentence Examples

[1120] 1. "What time is the next meeting?"

[1121] 2. "Tell me about your new project."

[1122] (Application example 1)

[1123] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1124] Currently, it is difficult for hearing-impaired employees to communicate with customers in noisy brick-and-mortar stores. It is also difficult for employees to clearly hear what customers are saying when they are wearing masks or in crowded environments. In these environments, an effective method for smoothly conducting conversations with customers is needed.

[1125] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1126] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into character data, means for returning the character data to the terminal, means for displaying the character data on a visual information display device, and means for the visual information display device to be used by employees of the physical store to communicate with customers. This allows employees of the physical store to visually confirm what customers are saying, enabling smoother communication.

[1127] A "means for accepting voice input" is a device or part of a system that collects voice data, recognizes voice through hardware such as a microphone, and processes it as digital data.

[1128] "Means for capturing voice input as audio data" refers to a device or process that records received audio in a digital data format and converts it into analyzable data.

[1129] "Means for transmitting audio data to a server" refers to a communication interface or protocol for transferring captured audio data to a remote server via a network.

[1130] "Means for converting voice data into text data" refers to the speech recognition technology or software used to analyze received voice data and generate a corresponding string of characters.

[1131] "Means for returning character data to the terminal" refers to the communication functions and protocols for returning character data generated by the server to the original terminal via the network.

[1132] "Means for displaying character data on a visual information display device" refers to a display device such as a display or AR glasses, and its control system, for visually presenting the character data returned to the terminal to the user.

[1133] A "visual information display device" is an electronic device that provides information visually to a user, and includes displays, AR glasses, head-mounted displays, etc.

[1134] A "physical store" is a physical space, a place where products and services are offered and where store staff and customers can communicate face-to-face.

[1135] A "personal digital assistant" is a portable electronic device capable of communicating voice and data, including smartphones and tablets.

[1136] This invention provides a system that converts voice input into text and displays it in real time when employees of a brick-and-mortar store communicate with customers. This system accepts voice input, converts it into text data, and displays it to employees on a visual information display device, enabling appropriate communication even in noisy environments or for those with hearing impairments.

[1137] Hardware and software used

[1138] This system requires the following hardware and software:

[1139] Hardware:

[1140] A microphone that accepts voice input (built into the visual information display device)

[1141] Visual information display devices (e.g., AR glasses, smart glasses)

[1142] Mobile information terminals (e.g. smartphones)

[1143] software:

[1144] Audio Capture Software

[1145] Communication Interface

[1146] Speech recognition engine (e.g. Whisper model)

[1147] Data Display Software

[1148] The server converts the voice data into text data using voice recognition technology, and the terminal receives the text data and displays it on a visual information display device.

[1149] Data processing and calculation methods

[1150] 1. Audio capture:

[1151] The user accepts voice input through the visual information display device. For example, a store employee wears smart glasses and converses with a customer.

[1152] 2. Sending audio data:

[1153] The device then sends the captured audio data to the server, where it encrypts the data using its communication function to ensure secure transmission.

[1154] 3. Speech recognition and conversion to text data:

[1155] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into corresponding text data.

[1156] 4. Returning character data:

[1157] The server returns the character data to the terminal, which displays the character data on a visual information display device.

[1158] 5. Character display:

[1159] Users can check text data in real time through the visual information display device, which allows them to visually recognize what the customer is saying and respond appropriately.

[1160] Specific examples

[1161] For example, if a customer says to a store employee, "Please explain this product to me," the smart glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the smart glasses as "Please explain this product to me." As a result, the store employee can see the text displayed on the visual information display device and understand the content of the conversation.

[1162] Example prompt for a generative AI model:

[1163] "Please tell me the specific steps to implement a system that transcribes conversations in a physical store in real time. Please provide a detailed explanation including voice capture, sending the voice data to the server, voice recognition, and displaying the text data."

[1164] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1165] Step 1:

[1166] A user wears a visual information display device (e.g., smart glasses) and talks with a customer.

[1167] Input: Customer voice

[1168] How it works: A microphone built into the visual display captures sound.

[1169] Output: Digital audio data

[1170] Step 2:

[1171] The terminal transmits the captured audio data to a server via a network.

[1172] Input: Digital audio data

[1173] How it works: The device's communications interface encrypts the audio data and sends it to the server.

[1174] Output: Audio data sent to the server

[1175] Step 3:

[1176] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into text data.

[1177] Input: Audio data transferred to the server

[1178] How it works: The server's speech recognition engine analyzes the audio data and generates corresponding text data.

[1179] Output: Character data

[1180] Step 4:

[1181] The server returns the generated character data to the terminal.

[1182] Input: Character data

[1183] Operation: The server's communication interface sends character data to the terminal.

[1184] Output: Character data sent to the terminal

[1185] Step 5:

[1186] The terminal displays the received character data on a visual information display device in real time.

[1187] Input: Character data sent to the terminal

[1188] Operation: The visual information display device displays the received character data on the display.

[1189] Output: Text data that can be visually confirmed by the user

[1190] Specific examples

[1191] Let's consider what happens when a customer asks an employee in a physical store, "Please explain this product to me."

[1192] Step 1: The microphone built into the visual display captures the audio, "Please describe this product."

[1193] Step 2: The device's communication interface encrypts this voice data and sends it to the server.

[1194] Step 3: The server's speech recognition engine (e.g., Whisper model) analyzes the voice data and generates text data such as "Please describe this product."

[1195] Step 4: The server's communication interface sends the character data to the terminal.

[1196] Step 5: The visual information display device displays the received text data on the screen, allowing the employee to confirm the content as text.

[1197] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1198] This invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts voice into text and analyzes and visually confirms the user's emotions.

[1199] In a face-to-face conversation

[1200] 1. The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[1201] 2. The device (AR glasses) captures the voice in real time. The voice of the person you are talking to is collected as digital audio data.

[1202] 3. The device temporarily stores the collected voice data and compresses and encrypts it.

[1203] 4. The device sends the compressed and encrypted audio data to the server. This communication uses a secure protocol.

[1204] 5. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1205] 6. The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1206] 7. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[1207] 8. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the terminal.

[1208] 9. The device (AR glasses) decodes the received text data and emotion data and displays it on the AR glasses display.

[1209] 10. Users can check the content of the conversation as well as the emotional state of the person they are talking to in real time.

[1210] Examples:

[1211] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server transcribes the voice, and at the same time, the emotion recognition engine analyzes whether the user is speaking in a serious tone, and the message "What time is the next meeting? (Emotion: Serious)" is displayed on the AR glasses' display. The user can visually confirm the content of the conversation and the emotion.

[1212] For telephone communication

[1213] 1. A user launches a calling app on their smartphone and makes a call to the other party. The user's voice and the other party's voice are captured in real time by the smartphone's microphone.

[1214] 2. The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[1215] 3. The device sends the compressed and encrypted audio data to the server, again using a secure protocol.

[1216] 4. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1217] 5. The server analyzes the voice data and converts it into text data, which is then stored in internal memory.

[1218] 6. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[1219] 7. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the device.

[1220] 8. The device (smartphone) decodes the received text data and emotion data and displays it on the call app screen.

[1221] 9. Users can check the content of the call as well as the emotional state of the other party in real time.

[1222] Examples:

[1223] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server transcribes the audio, and at the same time, an emotion recognition engine analyzes whether the user is speaking in an interested tone, and the message "Tell me about your new project (emotion: interest)" is displayed on the smartphone screen. The user can visually confirm the content of the call and the emotion.

[1224] This system improves the quality of face-to-face and telephone communication by capturing speech in real time, transcribing it, analyzing emotions, and displaying the results simultaneously. This process allows users to visually confirm both the content and emotions of the communication, enabling smoother dialogue.

[1225] The processing flow will be explained below.

[1226] In a face-to-face conversation

[1227] Step 1:

[1228] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[1229] Step 2:

[1230] The device (AR glasses) captures audio in real time, and the voice of the person you are talking to is collected as digital audio data.

[1231] Step 3:

[1232] The device temporarily stores the collected voice data in its internal memory and simultaneously compresses and encrypts it.

[1233] Step 4:

[1234] The device sends compressed and encrypted audio data to the server using a secure protocol.

[1235] Step 5:

[1236] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[1237] Step 6:

[1238] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1239] Step 7:

[1240] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[1241] Step 8:

[1242] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[1243] Step 9:

[1244] The device decodes the received text data and emotion data and displays it on the AR glasses' display.

[1245] Step 10:

[1246] Users can check the content of the conversation as well as the emotions of their conversation partner in real time.

[1247] For telephone communication

[1248] Step 1:

[1249] A user starts a calling app on their smartphone and makes a call to the other party. The voices of both parties are captured in real time by the smartphone's microphone during the call.

[1250] Step 2:

[1251] The device (smartphone) temporarily stores the captured audio data in its internal memory, and simultaneously compresses and encrypts it.

[1252] Step 3:

[1253] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[1254] Step 4:

[1255] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[1256] Step 5:

[1257] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1258] Step 6:

[1259] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[1260] Step 7:

[1261] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[1262] Step 8:

[1263] The device decodes the received text data and emotion data and displays them on the call app screen.

[1264] Step 9:

[1265] Users can see the content of the call as well as the emotions of the person they are talking to in real time.

[1266] In this way, a system is realized that converts voice communication into text and then analyzes and displays emotional data, improving the quality of face-to-face and telephone communication.

[1267] Example 2

[1268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1269] In modern voice communication, understanding the emotions of the person you're speaking to is extremely important. However, conventional systems only transcribe speech and are unable to analyze emotions, making it difficult for users to fully understand the content of the conversation. Furthermore, there was a lack of technology that could consistently perform real-time transcription and emotion analysis, preventing efficient communication support. This could result in a decline in the quality of the conversation and the risk of misunderstandings and breakdowns in communication.

[1270] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1271] In this invention, the server includes means for decoding voice data and converting it into text data, means for analyzing emotions from the voice data, and means for compressing and encrypting the text data and emotional data, thereby enabling real-time transcription and emotional analysis of voice communication.

[1272] "Voice input" refers to words or dialogue spoken by a user.

[1273] "Audio Data" means captured audio input stored in digital form.

[1274] "Compression" is the technique of reducing the size of data so that it can be stored and transmitted efficiently.

[1275] "Encryption" is the process of converting data into an unintelligible form for protection.

[1276] A "server" is a computer system responsible for analyzing and processing audio data.

[1277] "Decoding" is the technique of restoring encrypted data to its original form.

[1278] "Character data" is data in text format into which voice data has been converted.

[1279] "Emotion analysis" is a technology that identifies a speaker's emotions by analyzing the tone and pitch of a voice.

[1280] A "terminal" is a device that captures and displays audio data.

[1281] "Display" refers to the act of visually showing text and emotion data to the user.

[1282] A "head-mounted display" is a display device that is worn on the head.

[1283] A "personal digital assistant" is a portable device that captures and displays audio data.

[1284] The present invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts the voice into text and analyzes and visually confirms the user's emotions.

[1285] This system consists of a terminal that accepts voice input and captures the voice data, and a server that analyzes the voice data. Specifically, the following hardware and software are used:

[1286] Hardware

[1287] 1. Terminal: A device that captures and displays audio data. For face-to-face conversations, a head-mounted display is used, and for telephone communication, a personal digital assistant is used.

[1288] 2. Server: A computer system that decodes audio data, converts it to text data, and analyzes emotions.

[1289] software

[1290] 1. Speech recognition software: converts voice data into text data. For example, we will use Whisper.

[1291] 2. Emotion recognition engine: Identifies emotions by analyzing voice tone, pitch, etc.

[1292] Processing flow

[1293] In a face-to-face conversation

[1294] 1. The user puts on the head-mounted display and prepares to start a conversation with the other person. The display's microphone prepares for voice capture.

[1295] 2. The device (head-mounted display) captures audio in real time and temporarily stores it as digital audio data.

[1296] 3. The device compresses and then encrypts the collected audio data.

[1297] 4. The device sends the audio data to the server using a secure communication protocol such as TLS or HTTPS.

[1298] 5. The server decodes the received voice data and inputs it into the voice recognition software.

[1299] 6. The server analyzes the voice data and converts it into text data.

[1300] 7. The server uses an emotion recognition engine to analyze emotions from the voice data.

[1301] 8. The server re-compresses and encrypts the converted text data and emotion data and sends them back to the terminal.

[1302] 9. The device (head-mounted display) decodes the voice and emotion data and displays it on the display.

[1303] 10. The user visually confirms the displayed text data and emotional information.

[1304] For telephone communication

[1305] 1. The user launches the calling app on their mobile device and calls the person they want to call. The microphone on the mobile device prepares to capture audio.

[1306] 2. The device (mobile information terminal) captures the audio data in real time and temporarily stores it.

[1307] 3. The device compresses and encrypts the captured audio data.

[1308] 4. The device sends the audio data to the server using a secure communication protocol.

[1309] 5. The server decodes the received data and inputs it into the voice recognition software.

[1310] 6. The server analyzes the voice data and converts it into text data.

[1311] 7. The server analyzes the voice data using an emotion recognition engine to identify the emotion.

[1312] 8. The server re-compresses the converted text data and emotion data, encrypts them, and returns them to the terminal.

[1313] 9. The device (mobile information terminal) decodes the data and displays it in the calling application.

[1314] 10. The user visually confirms the text data and emotional information.

[1315] Specific examples

[1316] As an example of a prompt sentence, if a user says "What time is the next meeting?", the AR glasses will capture the voice and send it to the server. The server will convert the voice into text and analyze it with an emotion recognition engine, which will then display "What time is the next meeting? (Emotion: Serious)" on the AR glasses. The user can see this display in real time.

[1317] Furthermore, in the case of telephone communication, when a user says, "Tell me about your new project," the mobile information terminal captures the voice and sends it to the server. The server converts it into text data, which is displayed on the mobile information terminal as, "Tell me about your new project (emotion: interest)."

[1318] This system can improve the quality of face-to-face and telephone communication by transcribing speech in real time, analyzing emotions, and displaying the results.

[1319] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1320] In a face-to-face conversation

[1321] Step 1:

[1322] The user puts on the head-mounted display and prepares to start a conversation with the other person. The microphone of the head-mounted display prepares to capture voice.

[1323] Input: The user puts on the head-mounted display and begins interacting.

[1324] Output: The microphone is now operational and ready to capture audio.

[1325] Step 2:

[1326] The device (head-mounted display) captures surrounding sounds in real time, and the microphone collects the voice of the person you are talking to with high accuracy and temporarily stores it as digital audio data.

[1327] Input: Voice of the person you are talking to.

[1328] Output: Temporarily stored as digital audio data.

[1329] Step 3:

[1330] The terminal compresses the collected audio data and then encrypts it.

[1331] Input: Digital audio data.

[1332] Output: Compressed and encrypted audio data.

[1333] Step 4:

[1334] The device sends compressed and encrypted audio data to the server using secure communication protocols such as TLS and HTTPS.

[1335] Input: Compressed and encrypted audio data.

[1336] Output: Sent to the server.

[1337] Step 5:

[1338] The server decodes the received audio data and restores it to the original audio data.

[1339] Input: Encrypted audio data.

[1340] Output: Decoded audio data.

[1341] Step 6:

[1342] The server inputs the recovered voice data into voice recognition software (e.g., Whisper) and converts it into text data, which is then temporarily stored in internal memory.

[1343] Input: Decoded audio data.

[1344] Output: Character data.

[1345] Step 7:

[1346] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes the tone, pitch and other parameters of the voice to identify emotions.

[1347] Input: Decoded audio data.

[1348] Output: Emotion data.

[1349] Step 8:

[1350] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[1351] Input: Text data and emotion data.

[1352] Output: Compressed and encrypted text and emotion data.

[1353] Step 9:

[1354] The terminal (head-mounted display) decodes the voice and emotion data and displays it on the display.

[1355] Input: Compressed and encrypted character data and emotion data.

[1356] Output: Decoded character data and emotion data are displayed on the screen.

[1357] Step 10:

[1358] The user visually checks the displayed character data and emotion information.

[1359] Input: Character data and emotion data displayed on the display.

[1360] Output: User confirmation.

[1361] For telephone communication

[1362] Step 1:

[1363] The user launches a calling application on the mobile information terminal and makes a call to the other party. The microphone on the mobile information terminal prepares for voice capture.

[1364] Input: The user launches a calling app on a mobile information terminal and starts a call.

[1365] Output: The microphone is now operational and ready to capture audio.

[1366] Step 2:

[1367] The device (mobile information terminal) captures the voices of the user and the other party in real time, and temporarily stores them as digital voice data.

[1368] Input: Caller and user voice.

[1369] Output: Temporarily stored as digital audio data.

[1370] Step 3:

[1371] The device compresses the captured audio data and then encrypts it.

[1372] Input: Digital audio data.

[1373] Output: Compressed and encrypted audio data.

[1374] Step 4:

[1375] The device sends compressed and encrypted audio data to the server using a secure communication protocol.

[1376] Input: Compressed and encrypted audio data.

[1377] Output: Sent to the server.

[1378] Step 5:

[1379] The server decodes the received audio data and restores it to the original audio data.

[1380] Input: Encrypted audio data.

[1381] Output: Decoded audio data.

[1382] Step 6:

[1383] Similarly, the server inputs the voice data into voice recognition software and converts it into text data, which is then temporarily stored in internal memory.

[1384] Input: Decoded audio data.

[1385] Output: Character data.

[1386] Step 7:

[1387] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes parameters such as tone, pitch, speed, and volume of the voice to identify emotions.

[1388] Input: Decoded audio data.

[1389] Output: Emotion data.

[1390] Step 8:

[1391] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[1392] Input: Text data and emotion data.

[1393] Output: Compressed and encrypted text and emotion data.

[1394] Step 9:

[1395] The terminal (mobile information terminal) decodes the received text data and emotion data and displays them on the screen of the calling app.

[1396] Input: Compressed and encrypted character data and emotion data.

[1397] Output: Decoded character data and emotion data are displayed on the screen.

[1398] Step 10:

[1399] The user visually checks the displayed character data and emotion information.

[1400] Input: Character data and emotion data displayed on the screen.

[1401] Output: User confirmation.

[1402] (Application example 2)

[1403] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1404] In today's brick-and-mortar stores, it is difficult to accurately grasp a customer's intentions and emotions when dealing with customers, making improving customer satisfaction a challenge. If it were possible to instantly understand a customer's emotions on-site, more appropriate responses could be made. Furthermore, displaying voice communication as text in real time would clearly understand the customer's intentions and reduce misunderstandings. However, existing systems that achieve these goals are limited, and in many cases, problems include insufficient emotion analysis capabilities and insufficient system response speed.

[1405] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1406] In this invention, the server includes means for receiving voice input in real time, means for converting voice data into text data using voice recognition software, means for analyzing emotions from the voice data, and means for returning the analyzed emotional data and text data to the terminal. This makes it possible to convert conversations with customers into text in real time and to visually display the customer's emotions. This allows store clerks to instantly understand the customer's emotions and take appropriate action, contributing to improved customer satisfaction.

[1407] "Voice input" is the means by which human speech is received in digital form.

[1408] "Audio data" is data that represents audio as a digital signal.

[1409] A "server" is a computer system that processes information and manages data on a network.

[1410] "Character data" refers to text data converted from voice data using voice recognition technology.

[1411] "Terminal" means a device with input and output capabilities that can be directly operated by a user.

[1412] A "user" is a person who uses the system.

[1413] "Visual device" refers to a device that has the function of displaying visual information, and includes, for example, AR glasses.

[1414] A "mobile information terminal" is a portable information processing device that has wireless communication capabilities, and includes, for example, a smartphone.

[1415] "Means of emotion analysis" refers to technology that identifies a speaker's emotions based on parameters such as tone and pitch of the voice.

[1416] "Visual display means" refers to a device or method for displaying text or emotional information on a screen or the like.

[1417] An embodiment of the present invention will be described. An example is given in which a system that displays voice communication as text in real time and analyzes emotions and displays them visually is used in a physical store. A user uses this system to facilitate smoother conversations with customers.

[1418] System Program

[1419] The server processes audio data using multiple pieces of hardware and software. Specifically, it has the following configuration:

[1420] Hardware: The servers are equipped with high-performance processors and large amounts of memory, which allows for the processing of large amounts of voice data in real time. Smart glasses and smartphones are used as terminals.

[1421] Software: The server runs voice recognition software (e.g., Whisper) and an emotion recognition engine. The smart glasses and smartphones have a dedicated application installed to send and receive voice data.

[1422] Processing flow

[1423] When a user wears the smart glasses and begins talking to a customer, their voice is captured by the glasses' microphone. This voice data is temporarily stored in the smart glasses, compressed, encrypted, and sent to a server. On the server side, the voice data is decoded and converted into text data using voice recognition software. At the same time, an emotion recognition engine analyzes emotions from the voice data and generates emotion data.

[1424] The generated text and emotion data are compressed and encrypted again and sent back to the device. The smart glasses decode this data and display it on the screen. This allows the user to not only visually confirm the content of the conversation with the customer in text, but also understand the customer's emotions.

[1425] Specific examples

[1426] For example, if a customer says, "Tell me more about this product," the microphone in the glasses captures this voice and sends it to the server. This voice is converted into text data, and the emotion recognition engine analyzes the emotion of "interest." The smart glasses' display then displays text information such as "Tell me more about this product (emotion: interest)." This allows the user (store clerk) to understand the customer's interest and provide appropriate explanations.

[1427] Prompt Sentence Examples

[1428] The following is an example of a prompt to be input to a generative AI model that transcribes user speech in real time and then recognizes and displays emotions:

[1429] "When a customer says, 'Tell me more about this product,' transcribe that speech in real time and recognize and display the emotion 'interested.'"

[1430] This will significantly improve customer service in physical stores and increase customer satisfaction.

[1431] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1432] Step 1:

[1433] The user puts on the smart glasses and begins talking to the customer, whose voice is captured in real time by the smart glasses' microphone.

[1434] How it works: The smart glasses' built-in microphone collects audio digitally. The input to this process is the customer's voice, and the output is digital audio data.

[1435] Step 2:

[1436] The device (smart glasses) temporarily stores the captured audio data and compresses and encrypts it.

[1437] How it works: Audio data is stored in temporary storage and encrypted using an encryption algorithm such as AES. The input to this process is digital audio data, and the output is compressed and encrypted audio data.

[1438] Step 3:

[1439] The device (smart glasses) sends compressed and encrypted audio data to the server using a secure protocol (e.g., HTTPS).

[1440] Specific operation: Using the data transmission library, compressed and encrypted audio data is uploaded to the server. The input of this process is the compressed and encrypted audio data, and the output is the data transmitted to the server.

[1441] Step 4:

[1442] The server decodes the received voice data and converts it into text data using voice recognition software (e.g., Whisper).

[1443] Specific operation: Decodes the audio data using a decoding library and calls the speech recognition API to generate text data. The input of this process is the received audio data, and the output is the text data.

[1444] Step 5:

[1445] The server uses an emotion recognition engine to analyze emotions from the voice data.

[1446] Specific operation: The emotion recognition engine uses analysis algorithms to extract emotional parameters (e.g., tone, pitch, etc.) from speech and generates emotional data. The input of this process is speech data, and the output is emotional data.

[1447] Step 6:

[1448] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the terminal.

[1449] Specific operation: Text data and emotion data are compressed and encrypted using the compression library and encryption library, and then sent to the terminal using the data transmission library. The input of this process is character data and emotion data, and the output is compressed and encrypted character data and emotion data.

[1450] Step 7:

[1451] The device (smart glasses) decodes the received text data and emotion data and displays it on the screen.

[1452] Specific operation: Decode using the decoding library, and display text and emotions on the AR display using the display library. The input of this process is the compressed and encrypted character data and emotion data, and the output is the text and emotion data displayed on the smart glasses display.

[1453] Step 8:

[1454] The user can check the content of the conversation and the emotions of the person they are talking to through the display on the smart glasses.

[1455] Specific operation: The user visually checks the text and emotional information displayed on the smart glasses display and responds to the customer. The input of this process is the text displayed on the display and emotional data, and the output is the user's behavior.

[1456] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1457] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1458] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1459] [Fourth embodiment]

[1460] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1461] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1462] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1463] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1464] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1465] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1466] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1467] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1468] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1469] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1470] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1471] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1472] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] This invention relates to a system for displaying voice communication as text in real time. The system converts voice into text and allows users to visually confirm the text when communicating with a conversation partner in person or over the phone.

[1474] In a face-to-face conversation

[1475] 1. The user puts on the AR glasses and talks to the other person.

[1476] The AR glasses have built-in microphones that capture audio as users speak.

[1477] 2. The device (AR glasses) captures audio in real time.

[1478] The voices of the person you are speaking to are collected as digital voice data.

[1479] 3. The device sends the collected voice data to the server.

[1480] The communication function of the AR glasses is used to send audio data to the server.

[1481] 4. The server analyzes the voice data and converts it into text data.

[1482] The server uses voice recognition technology such as Whisper to convert the audio into text.

[1483] 5. The server returns the converted character data to the terminal.

[1484] The text data is then sent back to the AR glasses.

[1485] 6. The device displays the received text data on the AR display.

[1486] Users can view the content of the conversation in real time as text.

[1487] Examples:

[1488] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the AR glasses, where "What time is the next meeting?" is displayed on the display. The user can visually confirm the content.

[1489] For telephone communication

[1490] 1. The user launches the calling app on their smartphone and makes a call.

[1491] The smartphone will begin recording the audio of both parties during the call.

[1492] 2. The device (smartphone) captures the audio from both parties.

[1493] The audio of the call is collected as digital data.

[1494] 3. The device sends the captured audio data to the server.

[1495] The smartphone encrypts the voice data it collects and sends it to a server.

[1496] 4. The server analyzes the voice data and converts it into text data.

[1497] The dialogue is converted into text using voice recognition technology such as the Whisper model.

[1498] 5. The server returns the converted character data to the terminal.

[1499] The text data is sent to the smartphone.

[1500] 6. The device displays the received text data on the smartphone screen.

[1501] Users can view the contents of phone conversations as text in real time.

[1502] Examples:

[1503] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server converts the audio into text and sends it back to the smartphone, where "Tell me about your new project" is displayed on the screen. The user can visually confirm the content.

[1504] This system converts speech into text in real time during face-to-face and remote communication, enabling smooth conversations even for people with hearing impairments or in noisy environments. This allows users to visually check the transcribed data, improving the quality of communication.

[1505] The processing flow will be explained below.

[1506] In a face-to-face conversation

[1507] Step 1:

[1508] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[1509] Step 2:

[1510] The user's voice is captured in real time by the microphone in the AR glasses.

[1511] Step 3:

[1512] The device (AR glasses) temporarily stores the captured audio data and compresses and encrypts it.

[1513] Step 4:

[1514] The device sends compressed and encrypted audio data to the server using a secure protocol.

[1515] Step 5:

[1516] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1517] Step 6:

[1518] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1519] Step 7:

[1520] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[1521] Step 8:

[1522] The device (AR glasses) decodes the received text data and displays it on the AR glasses' display.

[1523] Step 9:

[1524] Users can see what their conversation partner is saying in real time.

[1525] For telephone communication

[1526] Step 1:

[1527] The user launches the calling app on their smartphone and calls the person they want to call.

[1528] Step 2:

[1529] The user's voice and the voice of the other party are captured in real time by the smartphone's microphone.

[1530] Step 3:

[1531] The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[1532] Step 4:

[1533] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[1534] Step 5:

[1535] The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1536] Step 6:

[1537] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1538] Step 7:

[1539] The server then compresses and encrypts the converted text data again and sends it back to the terminal.

[1540] Step 8:

[1541] The device (smartphone) decodes the received text data and displays it on the call app screen.

[1542] Step 9:

[1543] Users can view the contents of the call in real time as text.

[1544] This system allows face-to-face and telephone communications to be visually confirmed as text through a series of processes that captures voice in real time, converts it into text, and displays it.

[1545] Example 1

[1546] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1547] Conventional voice communication has the drawback of being difficult to communicate smoothly when the speaker is inaudible or when used in environments with a lot of background noise. Furthermore, people with hearing impairments and some elderly people have difficulty understanding spoken information, creating a need for a means to convert speech into text in real time. Furthermore, from the perspective of privacy protection, encryption technology is also needed to process voice data securely.

[1548] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1549] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into text data, means for returning the text data to the terminal, means for displaying the text data to the user, means for the means to use voice recognition technology to generate, transmit, and display visually identifiable text data in real time, and means for encrypting the voice data transmission and reception process. This allows users to visually confirm voice as text in real time, enabling smooth communication even in environments with a lot of background noise or for the hearing impaired. Furthermore, encryption of the voice data also enhances privacy protection.

[1550] The "means for accepting voice input" refers to a device or software function that captures the voice uttered by the user and environmental sounds and acquires the data necessary for subsequent processing.

[1551] "Means for capturing audio data" refers to the functionality of a device or software that digitally records audio input and converts it into a format suitable for further processing.

[1552] The "means for transmitting audio data to a server" refers to a device or software function that transfers captured audio data to a server via a communication line.

[1553] "Means for converting voice data into text data" refers to the functionality of a device or software within the server that converts voice data into text using voice recognition technology.

[1554] "Means for returning character data to the terminal" refers to the function of a device or software that sends the text data converted by the server back to the terminal via a communication line.

[1555] The "means for displaying character data to the user" refers to a device or software function that visually presents the returned text data to the user on a terminal display or the like.

[1556] "Means of using voice recognition technology" refers to technology or software functions for analyzing voice data and converting it into text with high accuracy.

[1557] "Encryption means" refers to the function of a device or software that encrypts data to maintain confidentiality during the communication of voice data and text data.

[1558] "Augmented reality glasses" are glasses-type devices that can be worn by the user to overlay virtual information (such as text data) onto the real world.

[1559] A "mobile phone" is a small portable communication terminal capable of voice calls and data communications.

[1560] This invention relates to a system that displays voice communication as text in real time, and is a system that allows a user to convert voice into text and visually confirm it when communicating with a conversation partner in person or over the phone.

[1561] This system is implemented as follows: First, a user uses either augmented reality glasses or a mobile phone. When a user wants to have a face-to-face conversation, the user puts on the augmented reality glasses and starts the conversation. The augmented reality glasses have a built-in microphone that captures the voice of the person they are talking to in real time. When a user wants to have a conversation over the phone, the voice data is captured using the mobile phone's calling app.

[1562] The device (augmented reality glasses or mobile phone) encodes the captured voice data and transmits it over a communication line to a server, where it is received and converted into text using voice recognition technology. Specifically, a voice recognition model such as OpenAI's Whisper is used.

[1563] Once the voice has been converted into text data, the server re-encrypts the text data and sends it back to the terminal via the communication line. The terminal then decrypts the received text data and displays it on a display so that the user can visually confirm it. This process occurs in real time, so the user can immediately see the contents of the conversation in text form.

[1564] The system also uses encryption technology to protect the privacy of users' voice data and converted text data, allowing users to use the system with peace of mind.

[1565] Specific examples

[1566] In a face-to-face conversation: When a user says, "What time is the next meeting?", the augmented reality glasses capture the voice and send it to the server. The server converts the voice into text data and sends it back to the augmented reality glasses. The user can see the text "What time is the next meeting?" displayed on the display inside the AR glasses.

[1567] In the case of telephone communication: When a user says "Tell me about your new project" on the phone, the mobile phone captures this voice and sends it to the server. The server converts the voice into text data and sends it back to the mobile phone. The user can see the text "Tell me about your new project" on the mobile phone screen.

[1568] Prompt Sentence Examples

[1569] 1. "What time is the next meeting?"

[1570] 2. "Tell me about your new project."

[1571] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1572] Step 1:

[1573] The device (augmented reality glasses or mobile phone) accepts voice input.

[1574] Input: User's voice.

[1575] How it works: In the case of augmented reality glasses, a microphone picks up what the user says and stores it as digital audio data. In the case of a mobile phone, the voice is recorded through a calling app.

[1576] Output: Captured as digital audio data and temporarily stored in memory.

[1577] Step 2:

[1578] The device sends the captured audio data to the server.

[1579] Input: Digital audio data.

[1580] What happens: Your device uses your internet connection to encrypt and send audio data to a server. Wi-Fi or cellular data is used.

[1581] Output: The encrypted audio data arrives at the server.

[1582] Step 3:

[1583] The server receives the audio data and begins analyzing it.

[1584] Input: Encrypted audio data.

[1585] Specific operation: The server decodes the received voice data and converts it into text using voice recognition technology (e.g., OpenAI's Whisper).

[1586] Output: The parsed text data is generated.

[1587] Step 4:

[1588] The server returns the converted character data to the terminal.

[1589] Input: Parsed text data.

[1590] Specific operation: The server encrypts the text data and sends it to the device over the communication line, again using Wi-Fi or mobile data.

[1591] Output: Encrypted text data arrives at the terminal.

[1592] Step 5:

[1593] The terminal decodes the received character data and displays it to the user.

[1594] Input: Encrypted text data.

[1595] Specific operation: The device decodes the received text data and displays it on the display in the case of augmented reality glasses, or on the screen in the case of a mobile phone.

[1596] Output: Text data that can be visually confirmed by the user.

[1597] Specific examples

[1598] Prompt Sentence Examples

[1599] 1. "What time is the next meeting?"

[1600] 2. "Tell me about your new project."

[1601] (Application example 1)

[1602] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1603] Currently, it is difficult for hearing-impaired employees to communicate with customers in noisy brick-and-mortar stores. It is also difficult for employees to clearly hear what customers are saying when they are wearing masks or in crowded environments. In these environments, an effective method for smoothly conducting conversations with customers is needed.

[1604] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1605] In this invention, the server includes means for accepting voice input, means for capturing the voice input as voice data, means for transmitting the voice data to the server, means for converting the voice data into character data, means for returning the character data to the terminal, means for displaying the character data on a visual information display device, and means for the visual information display device to be used by employees of the physical store to communicate with customers. This allows employees of the physical store to visually confirm what customers are saying, enabling smoother communication.

[1606] A "means for accepting voice input" is a device or part of a system that collects voice data, recognizes voice through hardware such as a microphone, and processes it as digital data.

[1607] "Means for capturing voice input as audio data" refers to a device or process that records received audio in a digital data format and converts it into analyzable data.

[1608] "Means for transmitting audio data to a server" refers to a communication interface or protocol for transferring captured audio data to a remote server via a network.

[1609] "Means for converting voice data into text data" refers to the speech recognition technology or software used to analyze received voice data and generate a corresponding string of characters.

[1610] "Means for returning character data to the terminal" refers to the communication functions and protocols for returning character data generated by the server to the original terminal via the network.

[1611] "Means for displaying character data on a visual information display device" refers to a display device such as a display or AR glasses, and its control system, for visually presenting the character data returned to the terminal to the user.

[1612] A "visual information display device" is an electronic device that provides information visually to a user, and includes displays, AR glasses, head-mounted displays, etc.

[1613] A "physical store" is a physical space, a place where products and services are offered and where store staff and customers can communicate face-to-face.

[1614] A "personal digital assistant" is a portable electronic device capable of communicating voice and data, including smartphones and tablets.

[1615] This invention provides a system that converts voice input into text and displays it in real time when employees of a brick-and-mortar store communicate with customers. This system accepts voice input, converts it into text data, and displays it to employees on a visual information display device, enabling appropriate communication even in noisy environments or for those with hearing impairments.

[1616] Hardware and software used

[1617] This system requires the following hardware and software:

[1618] Hardware:

[1619] A microphone that accepts voice input (built into the visual information display device)

[1620] Visual information display devices (e.g., AR glasses, smart glasses)

[1621] Mobile information terminals (e.g. smartphones)

[1622] software:

[1623] Audio Capture Software

[1624] Communication Interface

[1625] Speech recognition engine (e.g. Whisper model)

[1626] Data Display Software

[1627] The server converts the voice data into text data using voice recognition technology, and the terminal receives the text data and displays it on a visual information display device.

[1628] Data processing and calculation methods

[1629] 1. Audio capture:

[1630] The user accepts voice input through the visual information display device. For example, a store employee wears smart glasses and converses with a customer.

[1631] 2. Sending audio data:

[1632] The device then sends the captured audio data to the server, where it encrypts the data using its communication function to ensure secure transmission.

[1633] 3. Speech recognition and conversion to text data:

[1634] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into corresponding text data.

[1635] 4. Returning character data:

[1636] The server returns the character data to the terminal, which displays the character data on a visual information display device.

[1637] 5. Character display:

[1638] Users can check text data in real time through the visual information display device, which allows them to visually recognize what the customer is saying and respond appropriately.

[1639] Specific examples

[1640] For example, if a customer says to a store employee, "Please explain this product to me," the smart glasses capture the voice and send it to the server. The server converts the voice into text and sends it back to the smart glasses as "Please explain this product to me." As a result, the store employee can see the text displayed on the visual information display device and understand the content of the conversation.

[1641] Example prompt for a generative AI model:

[1642] "Please tell me the specific steps to implement a system that transcribes conversations in a physical store in real time. Please provide a detailed explanation including voice capture, sending the voice data to the server, voice recognition, and displaying the text data."

[1643] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1644] Step 1:

[1645] A user wears a visual information display device (e.g., smart glasses) and talks with a customer.

[1646] Input: Customer voice

[1647] How it works: A microphone built into the visual display captures sound.

[1648] Output: Digital audio data

[1649] Step 2:

[1650] The terminal transmits the captured audio data to a server via a network.

[1651] Input: Digital audio data

[1652] How it works: The device's communications interface encrypts the audio data and sends it to the server.

[1653] Output: Audio data sent to the server

[1654] Step 3:

[1655] The server analyzes the received voice data using a voice recognition engine (e.g., Whisper model) and converts it into text data.

[1656] Input: Audio data transferred to the server

[1657] How it works: The server's speech recognition engine analyzes the audio data and generates corresponding text data.

[1658] Output: Character data

[1659] Step 4:

[1660] The server returns the generated character data to the terminal.

[1661] Input: Character data

[1662] Operation: The server's communication interface sends character data to the terminal.

[1663] Output: Character data sent to the terminal

[1664] Step 5:

[1665] The terminal displays the received character data on a visual information display device in real time.

[1666] Input: Character data sent to the terminal

[1667] Operation: The visual information display device displays the received character data on the display.

[1668] Output: Text data that can be visually confirmed by the user

[1669] Specific examples

[1670] Let's consider what happens when a customer asks an employee in a physical store, "Please explain this product to me."

[1671] Step 1: The microphone built into the visual display captures the audio, "Please describe this product."

[1672] Step 2: The device's communication interface encrypts this voice data and sends it to the server.

[1673] Step 3: The server's speech recognition engine (e.g., Whisper model) analyzes the voice data and generates text data such as "Please describe this product."

[1674] Step 4: The server's communication interface sends the character data to the terminal.

[1675] Step 5: The visual information display device displays the received text data on the screen, allowing the employee to confirm the content as text.

[1676] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1677] This invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts voice into text and analyzes and visually confirms the user's emotions.

[1678] In a face-to-face conversation

[1679] 1. The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[1680] 2. The device (AR glasses) captures the voice in real time. The voice of the person you are talking to is collected as digital audio data.

[1681] 3. The device temporarily stores the collected voice data and compresses and encrypts it.

[1682] 4. The device sends the compressed and encrypted audio data to the server. This communication uses a secure protocol.

[1683] 5. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1684] 6. The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1685] 7. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[1686] 8. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the terminal.

[1687] 9. The device (AR glasses) decodes the received text data and emotion data and displays it on the AR glasses display.

[1688] 10. Users can check the content of the conversation as well as the emotional state of the person they are talking to in real time.

[1689] Examples:

[1690] When a user says, "What time is the next meeting?", the AR glasses capture the voice and send it to the server. The server transcribes the voice, and at the same time, the emotion recognition engine analyzes whether the user is speaking in a serious tone, and the message "What time is the next meeting? (Emotion: Serious)" is displayed on the AR glasses' display. The user can visually confirm the content of the conversation and the emotion.

[1691] For telephone communication

[1692] 1. A user launches a calling app on their smartphone and makes a call to the other party. The user's voice and the other party's voice are captured in real time by the smartphone's microphone.

[1693] 2. The device (smartphone) temporarily stores the captured audio data and compresses and encrypts it.

[1694] 3. The device sends the compressed and encrypted audio data to the server, again using a secure protocol.

[1695] 4. The server decodes the received voice data and inputs it into voice recognition software (e.g., Whisper).

[1696] 5. The server analyzes the voice data and converts it into text data, which is then stored in internal memory.

[1697] 6. The server uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, and other parameters of the voice to identify the user's emotions.

[1698] 7. The server compresses and encrypts the converted text data and analyzed emotion data and sends them back to the device.

[1699] 8. The device (smartphone) decodes the received text data and emotion data and displays it on the call app screen.

[1700] 9. Users can check the content of the call as well as the emotional state of the other party in real time.

[1701] Examples:

[1702] When a user says "Tell me about your new project" on the phone, the smartphone captures the audio and sends it to the server. The server transcribes the audio, and at the same time, an emotion recognition engine analyzes whether the user is speaking in an interested tone, and the message "Tell me about your new project (emotion: interest)" is displayed on the smartphone screen. The user can visually confirm the content of the call and the emotion.

[1703] This system improves the quality of face-to-face and telephone communication by capturing speech in real time, transcribing it, analyzing emotions, and displaying the results simultaneously. This process allows users to visually confirm both the content and emotions of the communication, enabling smoother dialogue.

[1704] The processing flow will be explained below.

[1705] In a face-to-face conversation

[1706] Step 1:

[1707] The user puts on the AR glasses and starts a conversation with the other person. The microphone in the AR glasses prepares to capture the surrounding sounds.

[1708] Step 2:

[1709] The device (AR glasses) captures audio in real time, and the voice of the person you are talking to is collected as digital audio data.

[1710] Step 3:

[1711] The device temporarily stores the collected voice data in its internal memory and simultaneously compresses and encrypts it.

[1712] Step 4:

[1713] The device sends compressed and encrypted audio data to the server using a secure protocol.

[1714] Step 5:

[1715] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[1716] Step 6:

[1717] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1718] Step 7:

[1719] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[1720] Step 8:

[1721] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[1722] Step 9:

[1723] The device decodes the received text data and emotion data and displays it on the AR glasses' display.

[1724] Step 10:

[1725] Users can check the content of the conversation as well as the emotions of their conversation partner in real time.

[1726] For telephone communication

[1727] Step 1:

[1728] A user starts a calling app on their smartphone and makes a call to the other party. The voices of both parties are captured in real time by the smartphone's microphone during the call.

[1729] Step 2:

[1730] The device (smartphone) temporarily stores the captured audio data in its internal memory, and simultaneously compresses and encrypts it.

[1731] Step 3:

[1732] The device sends compressed and encrypted audio data to the server, and this communication also uses a secure protocol.

[1733] Step 4:

[1734] The server receives and decodes the transmitted voice data, and inputs the decoded voice data into voice recognition software (e.g., Whisper).

[1735] Step 5:

[1736] The server analyzes the voice data and converts it into text data, which is then stored in the server's internal memory.

[1737] Step 6:

[1738] The server then uses an emotion recognition engine to analyze the user's emotions from the voice data. The emotion recognition engine analyzes the tone, pitch, speed, etc. of the voice to identify the user's emotions. This analyzed emotion data is stored together with the text data.

[1739] Step 7:

[1740] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the device.

[1741] Step 8:

[1742] The device decodes the received text data and emotion data and displays them on the call app screen.

[1743] Step 9:

[1744] Users can see the content of the call as well as the emotions of the person they are talking to in real time.

[1745] In this way, a system is realized that converts voice communication into text and then analyzes and displays emotional data, improving the quality of face-to-face and telephone communication.

[1746] Example 2

[1747] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1748] In modern voice communication, understanding the emotions of the person you're speaking to is extremely important. However, conventional systems only transcribe speech and are unable to analyze emotions, making it difficult for users to fully understand the content of the conversation. Furthermore, there was a lack of technology that could consistently perform real-time transcription and emotion analysis, preventing efficient communication support. This could result in a decline in the quality of the conversation and the risk of misunderstandings and breakdowns in communication.

[1749] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1750] In this invention, the server includes means for decoding voice data and converting it into text data, means for analyzing emotions from the voice data, and means for compressing and encrypting the text data and emotional data, thereby enabling real-time transcription and emotional analysis of voice communication.

[1751] "Voice input" refers to words or dialogue spoken by a user.

[1752] "Audio Data" means captured audio input stored in digital form.

[1753] "Compression" is the technique of reducing the size of data so that it can be stored and transmitted efficiently.

[1754] "Encryption" is the process of converting data into an unintelligible form for protection.

[1755] A "server" is a computer system responsible for analyzing and processing audio data.

[1756] "Decoding" is the technique of restoring encrypted data to its original form.

[1757] "Character data" is data in text format into which voice data has been converted.

[1758] "Emotion analysis" is a technology that identifies a speaker's emotions by analyzing the tone and pitch of a voice.

[1759] A "terminal" is a device that captures and displays audio data.

[1760] "Display" refers to the act of visually showing text and emotion data to the user.

[1761] A "head-mounted display" is a display device that is worn on the head.

[1762] A "personal digital assistant" is a portable device that captures and displays audio data.

[1763] The present invention relates to a system that converts voice communication into text in real time and recognizes the user's emotions. When a user communicates with a conversation partner face-to-face or over the phone, the system converts the voice into text and analyzes and visually confirms the user's emotions.

[1764] This system consists of a terminal that accepts voice input and captures the voice data, and a server that analyzes the voice data. Specifically, the following hardware and software are used:

[1765] Hardware

[1766] 1. Terminal: A device that captures and displays audio data. For face-to-face conversations, a head-mounted display is used, and for telephone communication, a personal digital assistant is used.

[1767] 2. Server: A computer system that decodes audio data, converts it to text data, and analyzes emotions.

[1768] software

[1769] 1. Speech recognition software: converts voice data into text data. For example, we will use Whisper.

[1770] 2. Emotion recognition engine: Identifies emotions by analyzing voice tone, pitch, etc.

[1771] Processing flow

[1772] In a face-to-face conversation

[1773] 1. The user puts on the head-mounted display and prepares to start a conversation with the other person. The display's microphone prepares for voice capture.

[1774] 2. The device (head-mounted display) captures audio in real time and temporarily stores it as digital audio data.

[1775] 3. The device compresses and then encrypts the collected audio data.

[1776] 4. The device sends the audio data to the server using a secure communication protocol such as TLS or HTTPS.

[1777] 5. The server decodes the received voice data and inputs it into the voice recognition software.

[1778] 6. The server analyzes the voice data and converts it into text data.

[1779] 7. The server uses an emotion recognition engine to analyze emotions from the voice data.

[1780] 8. The server re-compresses and encrypts the converted text data and emotion data and sends them back to the terminal.

[1781] 9. The device (head-mounted display) decodes the voice and emotion data and displays it on the display.

[1782] 10. The user visually confirms the displayed text data and emotional information.

[1783] For telephone communication

[1784] 1. The user launches the calling app on their mobile device and calls the person they want to call. The microphone on the mobile device prepares to capture audio.

[1785] 2. The device (mobile information terminal) captures the audio data in real time and temporarily stores it.

[1786] 3. The device compresses and encrypts the captured audio data.

[1787] 4. The device sends the audio data to the server using a secure communication protocol.

[1788] 5. The server decodes the received data and inputs it into the voice recognition software.

[1789] 6. The server analyzes the voice data and converts it into text data.

[1790] 7. The server analyzes the voice data using an emotion recognition engine to identify the emotion.

[1791] 8. The server re-compresses the converted text data and emotion data, encrypts them, and returns them to the terminal.

[1792] 9. The device (mobile information terminal) decodes the data and displays it in the calling application.

[1793] 10. The user visually confirms the text data and emotional information.

[1794] Specific examples

[1795] As an example of a prompt sentence, if a user says "What time is the next meeting?", the AR glasses will capture the voice and send it to the server. The server will convert the voice into text and analyze it with an emotion recognition engine, which will then display "What time is the next meeting? (Emotion: Serious)" on the AR glasses. The user can see this display in real time.

[1796] Furthermore, in the case of telephone communication, when a user says, "Tell me about your new project," the mobile information terminal captures the voice and sends it to the server. The server converts it into text data, which is displayed on the mobile information terminal as, "Tell me about your new project (emotion: interest)."

[1797] This system can improve the quality of face-to-face and telephone communication by transcribing speech in real time, analyzing emotions, and displaying the results.

[1798] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1799] In a face-to-face conversation

[1800] Step 1:

[1801] The user puts on the head-mounted display and prepares to start a conversation with the other person. The microphone of the head-mounted display prepares to capture voice.

[1802] Input: The user puts on the head-mounted display and begins interacting.

[1803] Output: The microphone is now operational and ready to capture audio.

[1804] Step 2:

[1805] The device (head-mounted display) captures surrounding sounds in real time, and the microphone collects the voice of the person you are talking to with high accuracy and temporarily stores it as digital audio data.

[1806] Input: Voice of the person you are talking to.

[1807] Output: Temporarily stored as digital audio data.

[1808] Step 3:

[1809] The terminal compresses the collected audio data and then encrypts it.

[1810] Input: Digital audio data.

[1811] Output: Compressed and encrypted audio data.

[1812] Step 4:

[1813] The device sends compressed and encrypted audio data to the server using secure communication protocols such as TLS and HTTPS.

[1814] Input: Compressed and encrypted audio data.

[1815] Output: Sent to the server.

[1816] Step 5:

[1817] The server decodes the received audio data and restores it to the original audio data.

[1818] Input: Encrypted audio data.

[1819] Output: Decoded audio data.

[1820] Step 6:

[1821] The server inputs the recovered voice data into voice recognition software (e.g., Whisper) and converts it into text data, which is then temporarily stored in internal memory.

[1822] Input: Decoded audio data.

[1823] Output: Character data.

[1824] Step 7:

[1825] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes the tone, pitch and other parameters of the voice to identify emotions.

[1826] Input: Decoded audio data.

[1827] Output: Emotion data.

[1828] Step 8:

[1829] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[1830] Input: Text data and emotion data.

[1831] Output: Compressed and encrypted text and emotion data.

[1832] Step 9:

[1833] The terminal (head-mounted display) decodes the voice and emotion data and displays it on the display.

[1834] Input: Compressed and encrypted character data and emotion data.

[1835] Output: Decoded character data and emotion data are displayed on the screen.

[1836] Step 10:

[1837] The user visually checks the displayed character data and emotion information.

[1838] Input: Character data and emotion data displayed on the display.

[1839] Output: User confirmation.

[1840] For telephone communication

[1841] Step 1:

[1842] The user launches a calling application on the mobile information terminal and makes a call to the other party. The microphone on the mobile information terminal prepares for voice capture.

[1843] Input: The user launches a calling app on a mobile information terminal and starts a call.

[1844] Output: The microphone is now operational and ready to capture audio.

[1845] Step 2:

[1846] The device (mobile information terminal) captures the voices of the user and the other party in real time, and temporarily stores them as digital voice data.

[1847] Input: Caller and user voice.

[1848] Output: Temporarily stored as digital audio data.

[1849] Step 3:

[1850] The device compresses the captured audio data and then encrypts it.

[1851] Input: Digital audio data.

[1852] Output: Compressed and encrypted audio data.

[1853] Step 4:

[1854] The device sends compressed and encrypted audio data to the server using a secure communication protocol.

[1855] Input: Compressed and encrypted audio data.

[1856] Output: Sent to the server.

[1857] Step 5:

[1858] The server decodes the received audio data and restores it to the original audio data.

[1859] Input: Encrypted audio data.

[1860] Output: Decoded audio data.

[1861] Step 6:

[1862] Similarly, the server inputs the voice data into voice recognition software and converts it into text data, which is then temporarily stored in internal memory.

[1863] Input: Decoded audio data.

[1864] Output: Character data.

[1865] Step 7:

[1866] The server uses an emotion recognition engine to analyze emotions from the voice data, which analyzes parameters such as tone, pitch, speed, and volume of the voice to identify emotions.

[1867] Input: Decoded audio data.

[1868] Output: Emotion data.

[1869] Step 8:

[1870] The server re-compresses and encrypts the converted character data and emotion data and sends them back to the terminal.

[1871] Input: Text data and emotion data.

[1872] Output: Compressed and encrypted text and emotion data.

[1873] Step 9:

[1874] The terminal (mobile information terminal) decodes the received text data and emotion data and displays them on the screen of the calling app.

[1875] Input: Compressed and encrypted character data and emotion data.

[1876] Output: Decoded character data and emotion data are displayed on the screen.

[1877] Step 10:

[1878] The user visually checks the displayed character data and emotion information.

[1879] Input: Character data and emotion data displayed on the screen.

[1880] Output: User confirmation.

[1881] (Application example 2)

[1882] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1883] In today's brick-and-mortar stores, it is difficult to accurately grasp a customer's intentions and emotions when dealing with customers, making improving customer satisfaction a challenge. If it were possible to instantly understand a customer's emotions on-site, more appropriate responses could be made. Furthermore, displaying voice communication as text in real time would clearly understand the customer's intentions and reduce misunderstandings. However, existing systems that achieve these goals are limited, and in many cases, problems include insufficient emotion analysis capabilities and insufficient system response speed.

[1884] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1885] In this invention, the server includes means for receiving voice input in real time, means for converting voice data into text data using voice recognition software, means for analyzing emotions from the voice data, and means for returning the analyzed emotional data and text data to the terminal. This makes it possible to convert conversations with customers into text in real time and to visually display the customer's emotions. This allows store clerks to instantly understand the customer's emotions and take appropriate action, contributing to improved customer satisfaction.

[1886] "Voice input" is the means by which human speech is received in digital form.

[1887] "Audio data" is data that represents audio as a digital signal.

[1888] A "server" is a computer system that processes information and manages data on a network.

[1889] "Character data" refers to text data converted from voice data using voice recognition technology.

[1890] "Terminal" means a device with input and output capabilities that can be directly operated by a user.

[1891] A "user" is a person who uses the system.

[1892] "Visual device" refers to a device that has the function of displaying visual information, and includes, for example, AR glasses.

[1893] A "mobile information terminal" is a portable information processing device that has wireless communication capabilities, and includes, for example, a smartphone.

[1894] "Means of emotion analysis" refers to technology that identifies a speaker's emotions based on parameters such as tone and pitch of the voice.

[1895] "Visual display means" refers to a device or method for displaying text or emotional information on a screen or the like.

[1896] An embodiment of the present invention will be described. An example is given in which a system that displays voice communication as text in real time and analyzes emotions and displays them visually is used in a physical store. A user uses this system to facilitate smoother conversations with customers.

[1897] System Program

[1898] The server processes audio data using multiple pieces of hardware and software. Specifically, it has the following configuration:

[1899] Hardware: The servers are equipped with high-performance processors and large amounts of memory, which allows for the processing of large amounts of voice data in real time. Smart glasses and smartphones are used as terminals.

[1900] Software: The server runs voice recognition software (e.g., Whisper) and an emotion recognition engine. The smart glasses and smartphones have a dedicated application installed to send and receive voice data.

[1901] Processing flow

[1902] When a user wears the smart glasses and begins talking to a customer, their voice is captured by the glasses' microphone. This voice data is temporarily stored in the smart glasses, compressed, encrypted, and sent to a server. On the server side, the voice data is decoded and converted into text data using voice recognition software. At the same time, an emotion recognition engine analyzes emotions from the voice data and generates emotion data.

[1903] The generated text and emotion data are compressed and encrypted again and sent back to the device. The smart glasses decode this data and display it on the screen. This allows the user to not only visually confirm the content of the conversation with the customer in text, but also understand the customer's emotions.

[1904] Specific examples

[1905] For example, if a customer says, "Tell me more about this product," the microphone in the glasses captures this voice and sends it to the server. This voice is converted into text data, and the emotion recognition engine analyzes the emotion of "interest." The smart glasses' display then displays text information such as "Tell me more about this product (emotion: interest)." This allows the user (store clerk) to understand the customer's interest and provide appropriate explanations.

[1906] Prompt Sentence Examples

[1907] The following is an example of a prompt to be input to a generative AI model that transcribes user speech in real time and then recognizes and displays emotions:

[1908] "When a customer says, 'Tell me more about this product,' transcribe that speech in real time and recognize and display the emotion 'interested.'"

[1909] This will significantly improve customer service in physical stores and increase customer satisfaction.

[1910] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1911] Step 1:

[1912] The user puts on the smart glasses and begins talking to the customer, whose voice is captured in real time by the smart glasses' microphone.

[1913] How it works: The smart glasses' built-in microphone collects audio digitally. The input to this process is the customer's voice, and the output is digital audio data.

[1914] Step 2:

[1915] The device (smart glasses) temporarily stores the captured audio data and compresses and encrypts it.

[1916] How it works: Audio data is stored in temporary storage and encrypted using an encryption algorithm such as AES. The input to this process is digital audio data, and the output is compressed and encrypted audio data.

[1917] Step 3:

[1918] The device (smart glasses) sends compressed and encrypted audio data to the server using a secure protocol (e.g., HTTPS).

[1919] Specific operation: Using the data transmission library, compressed and encrypted audio data is uploaded to the server. The input of this process is the compressed and encrypted audio data, and the output is the data transmitted to the server.

[1920] Step 4:

[1921] The server decodes the received voice data and converts it into text data using voice recognition software (e.g., Whisper).

[1922] Specific operation: Decodes the audio data using a decoding library and calls the speech recognition API to generate text data. The input of this process is the received audio data, and the output is the text data.

[1923] Step 5:

[1924] The server uses an emotion recognition engine to analyze emotions from the voice data.

[1925] Specific operation: The emotion recognition engine uses analysis algorithms to extract emotional parameters (e.g., tone, pitch, etc.) from speech and generates emotional data. The input of this process is speech data, and the output is emotional data.

[1926] Step 6:

[1927] The server then compresses and encrypts the converted text data and analyzed emotion data again and sends them back to the terminal.

[1928] Specific operation: Text data and emotion data are compressed and encrypted using the compression library and encryption library, and then sent to the terminal using the data transmission library. The input of this process is character data and emotion data, and the output is compressed and encrypted character data and emotion data.

[1929] Step 7:

[1930] The device (smart glasses) decodes the received text data and emotion data and displays it on the screen.

[1931] Specific operation: Decode using the decoding library, and display text and emotions on the AR display using the display library. The input of this process is the compressed and encrypted character data and emotion data, and the output is the text and emotion data displayed on the smart glasses display.

[1932] Step 8:

[1933] The user can check the content of the conversation and the emotions of the person they are talking to through the display on the smart glasses.

[1934] Specific operation: The user visually checks the text and emotional information displayed on the smart glasses display and responds to the customer. The input of this process is the text displayed on the display and emotional data, and the output is the user's behavior.

[1935] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1936] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1937] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1938] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1939] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1940] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1941] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1942] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1943] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1944] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1945] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1946] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1947] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1948] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1949] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1950] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1951] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1952] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1953] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1954] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1955] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1956] The following is further disclosed regarding the above embodiment.

[1957] (Claim 1)

[1958] means for accepting voice input;

[1959] means for capturing the voice input as voice data;

[1960] means for transmitting the voice data to a server;

[1961] means for converting the voice data into character data;

[1962] means for returning the character data to a terminal;

[1963] means for displaying said character data to a user.

[1964] (Claim 2)

[1965] 10. The system of claim 1, wherein the means for capturing audio comprises AR glasses.

[1966] (Claim 3)

[1967] 10. The system of claim 1, wherein the means for capturing audio comprises a smartphone.

[1968] "Example 1"

[1969] (Claim 1)

[1970] means for accepting voice input;

[1971] means for capturing the voice input as voice data;

[1972] means for transmitting the voice data to a server;

[1973] means for converting the voice data into character data;

[1974] means for returning the character data to a terminal;

[1975] means for displaying the character data to a user;

[1976] means for using speech recognition technology to generate, transmit, and display visually perceptible text data in real time;

[1977] A system including means for encrypting the transmission and reception process of voice data.

[1978] (Claim 2)

[1979] 10. The system of claim 1, wherein the means for capturing audio comprises augmented reality glasses.

[1980] (Claim 3)

[1981] 10. The system of claim 1, wherein the means for capturing audio comprises a mobile phone.

[1982] "Application Example 1"

[1983] (Claim 1)

[1984] means for accepting voice input;

[1985] means for capturing the voice input as voice data;

[1986] means for transmitting the voice data to a server;

[1987] means for converting the voice data into character data;

[1988] means for returning the character data to a terminal;

[1989] means for displaying the character data on a visual information display device;

[1990] A system in which the visual information display device includes a means for use by store employees to communicate with customers.

[1991] (Claim 2)

[1992] 10. The system of claim 1, wherein the means for capturing audio includes a visual information display device.

[1993] (Claim 3)

[1994] 10. The system of claim 1, wherein the means for capturing audio comprises a personal digital assistant.

[1995] "Example 2: Combining Emotion Engines"

[1996] (Claim 1)

[1997] means for accepting voice input;

[1998] means for capturing the voice input as voice data;

[1999] means for compressing and encrypting said audio data;

[2000] means for transmitting the voice data to a server;

[2001] means for decoding the voice data and converting it into character data;

[2002] means for analyzing emotions from the voice data;

[2003] means for compressing and encrypting the character data and emotion data;

[2004] means for returning the character data and emotion data to a terminal;

[2005] The system includes means for displaying the character data and emotion data to a user.

[2006] (Claim 2)

[2007] 10. The system of claim 1, wherein the means for capturing audio includes a head-mounted display.

[2008] (Claim 3)

[2009] 10. The system of claim 1, wherein the means for capturing audio comprises a personal digital assistant.

[2010] "Application example 2 when combining emotion engines"

[2011] (Claim 1)

[2012] means for accepting voice input;

[2013] means for capturing the voice input as voice data;

[2014] means for transmitting the voice data to a server;

[2015] means for converting the voice data into character data;

[2016] means for returning the character data to a terminal;

[2017] means for displaying the character data to a user;

[2018] means for analyzing emotions from the voice data;

[2019] means for returning the analyzed emotion data to a terminal;

[2020] means for displaying the analyzed emotion data to a user.

[2021] (Claim 2)

[2022] 10. The system of claim 1, wherein the means for capturing audio includes a visual device.

[2023] (Claim 3)

[2024] 10. The system of claim 1, wherein the means for capturing audio comprises a personal digital assistant. [Explanation of symbols]

[2025] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for accepting voice input; means for capturing the voice input as voice data; means for transmitting the voice data to a server; means for converting the voice data into character data; means for returning the character data to a terminal; means for displaying said character data to a user.

2. 10. The system of claim 1, wherein the means for capturing audio comprises AR glasses.

3. 10. The system of claim 1, wherein the means for capturing audio comprises a smartphone.

Images