system

Abstract

Provide a system. [Solution] means for receiving voice input from a user; means for converting the voice input into text data; means for transmitting the converted text data to a server; means for collecting information based on the received text data; A means for generating the collected information as a document; a means for converting the generated material into emotive speech; means for playing the audio; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Researching information and preparing materials takes time and effort, and a high level of skill is required to effectively present the materials. Therefore, there is a need to provide a system that supports efficient and effective information gathering and material preparation, and also allows for easy and high-quality presentations. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to a server, means for collecting information based on the received text data, means for generating materials from the collected information, means for converting the generated materials into emotive speech, and means for playing back the speech. This system allows users to efficiently conduct information research and create materials through voice input, and the generated materials are presented in emotive speech, enabling high-quality information communication.

[0006] "Voice input" refers to instructions or questions from the user via voice.

[0007] "Text data" refers to data that has been converted from voice input into text information.

[0008] A "server" refers to a computer system that provides various services and data over a network.

[0009] "Means of collecting information" refers to the process of searching for external information based on specific text data and collecting the necessary data.

[0010] "Means of generating materials" refers to the process of creating organized and structured documents and presentation materials based on collected information.

[0011] "Emotionally rich voice" refers to voice data that has natural intonation and emotional expression.

[0012] "Means for playing audio" refers to speakers or software that outputs the generated audio data. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a system that allows a user to request information research or the creation of materials through voice input, and then presents the created materials in an expressive voice. Specific embodiments of the present invention will be described below.

[0035] System Configuration

[0036] This system is realized by the cooperation of the user terminal and the server. The user terminal mainly handles voice input and processing, while the server is responsible for collecting information and creating materials.

[0037] Program operation overview

[0038] 1. Acquiring voice input

[0039] The user issues voice instructions to the device, such as "Research the world economy in 2023 and create a document."

[0040] 2. Voice to text conversion

[0041] The device receives voice input and converts it to text using speech recognition software, which processes the voice data as text.

[0042] 3. Sending text data to the server

[0043] The device then sends the converted text data to the server, using an appropriate API or similar.

[0044] 4. Information gathering and document preparation

[0045] The server uses generative AI based on the received text data to collect information, for example, by retrieving necessary information from external databases or the internet and creating documents based on that information.

[0046] 5. Conversion to emotionally rich speech

[0047] The server converts the generated material into emotive speech using text-to-speech (TTS) technology, which translates the material into natural intonation.

[0048] 6. Sending audio data to the device and playing it back

[0049] After generating the audio data, the server transmits it to the terminal, which then plays the received audio data in a format that presents it to the user.

[0050] Specific examples

[0051] Example 1: Requesting an economic report

[0052] 1. The user says, "Research the world economy in 2023 and create a document."

[0053] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0054] 3. The device sends this text data to the server.

[0055] 4. The server uses generative AI to collect the latest information on the global economy in 2023 and create a report.

[0056] 5. The server generates an emotive voice based on this report, for example, "The global economy in 2023 will be very unstable..."

[0057] 6. The server sends the audio file to the terminal, which plays it and reports it to the user.

[0058] In this way, by utilizing this system, users can easily request information research and document preparation, and receive the results in high-quality audio, enabling efficient and effective information gathering, document preparation, and presentations.

[0059] The processing flow will be explained below.

[0060] Step 1:

[0061] The user speaks to the device, for example, "Research the world economy in 2023 and create a document."

[0062] Step 2:

[0063] The device receives the user's voice input through a microphone and then uses speech recognition software to convert the received speech into text data.

[0064] Step 3:

[0065] The device sends the converted text data to the server, using an appropriate API to send the text data to the server over the network.

[0066] Step 4:

[0067] The server receives the text data and passes it to the generation AI, which then collects the necessary information based on the text data and creates materials. For example, it may collect information about the world economy in 2023 from external databases or the internet.

[0068] Step 5:

[0069] The server creates documents based on the collected information, which are organized in text format and clearly describe the content to be provided to users.

[0070] Step 6:

[0071] The server converts the created materials into emotionally rich speech, using text-to-speech (TTS) technology to generate speech data incorporating natural intonation and emotional expressions.

[0072] Step 7:

[0073] The server then sends the generated voice data to the device, which is also sent via the network via the API.

[0074] Step 8:

[0075] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[0076] This series of processing steps allows users to efficiently request information research and document creation, and receive the results in an expressive voice. This system significantly reduces users' time and effort, and enables high-quality presentations.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] In recent years, there has been a demand for more efficient information research and document creation, but the time and effort required for these tasks remains a challenge. In particular, there is a demand for systems that can efficiently and automatically perform a series of tasks, including collecting information using voice input, generating documents, and then presenting those documents with expressive voice. However, conventional systems require some of these processes to be performed manually, which can result in reduced efficiency.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text information, and means for transmitting the converted text information to a data processing device, thereby enabling efficient information gathering and document generation based on the voice input, and the results to be presented in an expressive voice.

[0082] "User" refers to a person who requests information research or document creation through voice input.

[0083] "Voice input" refers to the voice that a user speaks into a terminal, and is a means for making requests for information research or document creation by voice.

[0084] "Text information" refers to text data converted from voice input using voice recognition technology.

[0085] "Data processing device" refers to a computer system that receives text information and performs processes such as information collection, document creation, and audio data generation.

[0086] "Data" refers to information such as documents, numbers, statistics, and images obtained as a result of information gathering, and serves as the basis for creating materials.

[0087] "Document" refers to a material generated based on information collected by a data processing device and expressed in text format.

[0088] "Audio data" refers to data in which a document has been converted into expressive audio, and refers to an audio file that is output by a playback device.

[0089] A "playback device" refers to equipment or software that allows users to listen to audio data, and includes, for example, speakers and media players.

[0090] This invention is a system in which a user can request information research or document creation through voice input, and the created document is then presented in an expressive voice. This system is realized mainly through the cooperation of a user terminal and a server.

[0091] System Configuration

[0092] This system consists of the following components:

[0093] User Device: A device that receives voice input and converts it into text. Uses good voice recognition software (e.g., Google® Speech-to-Text API).

[0094] Server: A device that collects information, generates documents, and generates emotive voice data based on text information. This uses a generative AI model (e.g., OpenAI (registered trademark) GPT-4 (registered trademark)) and speech synthesis technology (e.g., Amazon Polly).

[0095] Communication method: API communication (e.g., HTTP POST) is used to send and receive data between the user terminal and the server.

[0096] Processing flow

[0097] 1. Acquisition of voice input: The user speaks into the device to request specific research or document creation. The device uses the built-in microphone to capture the voice.

[0098] 2. Speech-to-text conversion: The device uses the Google Speech-to-Text API to convert speech into text, turning the speech data into processable text.

[0099] 3. Sending text data to the server: The device sends the converted text data to the server via an HTTP POST request, which includes the destination URL and authentication information.

[0100] 4. Information collection and document creation: The server analyzes the received text data and uses a generative AI model (GPT-4) to collect the necessary information and create documents. The necessary information is collected from external databases and the Internet.

[0101] 5. Conversion to emotive speech: The server sends the generated material to Amazon Polly, which converts it into natural, emotive speech data.

[0102] 6. Sending and playing back audio data to the terminal: The server sends the generated audio file to the terminal, and the terminal plays back the audio data in a way that presents it to the user.

[0103] Specific examples

[0104] Example 1: Requesting an economic report

[0105] 1. The user says, "Research the world economy in 2023 and create a document."

[0106] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0107] 3. The device sends this text data to the server.

[0108] 4. The server uses a generative AI model (GPT-4) to collect the latest information on the global economy in 2023 and create a report.

[0109] 5. The server generates emotive audio based on this report, creating a presentation audio with the following content, for example, "The global economy in 2023 will be extremely unstable..."

[0110] 6. The server sends the audio data to the terminal, which plays it and reports it to the user.

[0111] Prompt Sentence Examples

[0112] "Research and write a detailed report on the global economy in 2023. Then, turn that report into an audio presentation."

[0113] "Gather information about the latest impacts of climate change, create materials and give an emotive presentation."

[0114] By utilizing this system, users can efficiently conduct information research, create documents, and make presentations.

[0115] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0116] Step 1:

[0117] The user speaks to the device to request specific research or document creation. The input is the user's voice, such as "Research the world economy in 2023 and create a document." The device picks up this voice using its built-in microphone.

[0118] Step 2:

[0119] The device sends the acquired voice data to the Google Speech-to-Text API and converts the voice into text information. The input is the voice data acquired in step 1, and text information is obtained by sending this voice data to the API. The output is text information in text format.

[0120] Specific behavior:

[0121] 1. Obtain the audio file recorded by the device.

[0122] 2. Send the acquired audio file to the Google Speech-to-Text API.

[0123] 3. Save the text information returned as an API response.

[0124] Step 3:

[0125] The terminal sends the converted text information to the server via an HTTP POST request. The input is the text information obtained in step 2, and an HTTP request is created to send this information to the server. The output is a response indicating that the request was successful.

[0126] Specific behavior:

[0127] 1. Convert the text information into JSON format and set it as the payload of the HTTP POST request.

[0128] 2. Set authentication information and destination URL in the request header.

[0129] 3. Send an HTTP request to the server and receive a response.

[0130] Step 4:

[0131] The server analyzes the received text information, collects the necessary information using a generative AI model (e.g., GPT-4), and generates the document. The input is the text information sent in step 3, and based on this information, it sends a prompt to the AI ​​model. The output is the generated document.

[0132] Specific behavior:

[0133] 1. The server analyzes the received text information.

[0134] 2. Generate a prompt for GPT-4 to begin collecting information.

[0135] 3. Create materials (documents) based on the returned information.

[0136] Step 5:

[0137] The server sends the created material to Amazon Polly, which converts it into natural, emotive speech data. The input is the material generated in step 4, which is sent to a text-to-speech API to be converted into speech. The output is an audio file.

[0138] Specific behavior:

[0139] 1. Save the generated material in text format.

[0140] 2. Send this text data to the Amazon Polly API.

[0141] 3. Save the audio file (e.g., MP3 format) returned as an API response.

[0142] Step 6:

[0143] The server sends the generated audio file to the terminal. The terminal receives this audio data and plays it in a form that is presented to the user. The input is the audio file generated in step 5, which is sent from the server to the terminal. The output is the playback of the audio file.

[0144] Specific behavior:

[0145] 1. The server sends the audio file in an HTTP response.

[0146] 2. The device saves the received audio file locally.

[0147] 3. Play the saved audio file in a media player and let the user listen to it.

[0148] Through this series of processes, the user can automatically conduct research based on the content specified by voice, create materials, and receive the content in an expressive voice.

[0149] (Application example 1)

[0150] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0151] In conventional self-driving vehicles, when a user requests information by voice, it is difficult to search for information or generate materials in real time. It is also difficult to present the acquired information in an expressive voice. As a result, users often experience inconvenience because they are unable to immediately obtain useful information about their destination or current situation. The present invention aims to solve these problems and provide a system that enables real-time, expressive voice presentations in self-driving vehicles.

[0152] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0153] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into expressive voice, means for use in an autonomous vehicle, means for providing information and generating materials in real time, means for generating materials using a generative AI model, and means for transmitting and playing voice to a terminal in the autonomous vehicle. This enables a user to obtain necessary information in real time in an autonomous vehicle and receive the information in expressive voice.

[0154] "Means for receiving voice input" refers to a device or software that recognizes the voice uttered by the user and inputs the voice data into the system.

[0155] The "means for converting speech input into text data" refers to a device or software that analyzes received speech data and outputs the resulting text data.

[0156] The "means for transmitting the converted text data to the server" refers to a communication device or software for sending the text data to the server via a network.

[0157] "Information gathering means" refers to a device or software that searches and retrieves the required information from a server or other database.

[0158] "Means for generating materials" refers to devices or software that process collected information and compile it into a format that is easy for users to use.

[0159] The "means for converting into emotionally rich speech" is a device or software that generates speech with natural intonation including emotional expressions based on the generated text data.

[0160] The "means for reproducing audio" is a device such as a speaker for reproducing the generated audio data as actual sound.

[0161] "In-vehicle means" means any device or software that is installed and operates within an automated vehicle.

[0162] "Means for providing information and generating materials in real time" refers to a device or software for instantly collecting information and generating materials in response to user instructions.

[0163] A "means for generating materials using a generative AI model" is a device or software that uses advanced artificial intelligence to collect information and generate materials.

[0164] The "means for transmitting audio to a terminal and playing it back" refers to a device or software that transmits audio data generated by a server to a terminal and plays back the audio on the terminal.

[0165] The present invention is a system that provides real-time information within an autonomous vehicle, generates materials, and reports them to the user in an expressive voice. The system receives voice input, converts the content into text data, and sends it to a server. The server collects information based on the received text data and generates materials using a generative AI model. The generated materials are converted into expressive voice, and the voice data is sent to a terminal. Finally, the terminal plays the received voice data for the user.

[0166] About program processing

[0167] 1. Acquiring voice input:

[0168] The device inside the car captures the user's voice. This uses voice recognition software (e.g., Google Cloud Speech-to-Text) to capture the user's voice as digital data.

[0169] 2. Voice to text conversion:

[0170] The captured voice data is analyzed and converted into corresponding text data. At this stage, speech recognition software converts speech into text.

[0171] 3. Sending text data:

[0172] The converted text data is sent to the server using the vehicle's communication module, where a RESTful API is used as the communication method.

[0173] 4. Information gathering and data generation:

[0174] The server uses a generative AI model (such as OpenAI's GPT-4) to collect information based on the received text data. Specifically, it searches external databases and information on the internet to collect and analyze the necessary data.

[0175] Based on the collected information, materials are generated. A generative AI model is used to generate these materials.

[0176] 5. Emotional voice conversion:

[0177] The material is converted into emotive audio using text-to-speech (TTS) technology, such as the TTS capabilities of Azure® Cognitive Services.

[0178] 6. Playing audio data:

[0179] The server transmits the generated voice data to an in-vehicle terminal, which then plays the voice data through a speaker.

[0180] Specific examples

[0181] For example, suppose a user speaks to an in-car device, saying, "Tell me about recommended restaurants near the next rest stop." This voice input is converted to text using Google Cloud Speech-to-Text and sent to the server. The server uses OpenAI's GPT-4 to collect information about recommended restaurants near the next rest stop from the internet. The collected information is compiled into a document, and the content is converted into emotive speech using Azure Cognitive Services' TTS technology. Finally, the speech is sent to the in-car device and played over the speaker. An example of a prompt sentence is, "Tell me about recommended restaurants near the next rest stop." This system allows users to obtain the information they need in real time and receive it in emotive speech.

[0182] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0183] Step 1:

[0184] The user inputs voice into the terminal inside the autonomous vehicle. Specifically, they speak through the microphone, saying, "Tell me about a recommended restaurant near the next rest stop." This voice data is captured by the terminal. The input is the user's voice, and the output is analog voice data.

[0185] Step 2:

[0186] The device receives voice input and converts the voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Specifically, the voice recognition engine analyzes the sound waveform and generates a corresponding string of characters. The input is voice data, and the output is text data.

[0187] Step 3:

[0188] The device sends the content converted into text data to the server. A RESTful API is used as the communication method, and the text data is sent as an HTTP POST request. Specifically, the device sends the generated text data to the server in JSON format. The input is text data, and the output is data sent to the server.

[0189] Step 4:

[0190] The server uses a generative AI model (such as OpenAI's GPT-4) to collect the necessary information based on the received text data. Specifically, it searches for and retrieves relevant information from the internet and internal databases. The input is text data, and the output is the collected information.

[0191] Step 5:

[0192] The server generates documents based on the collected information. Here too, a generative AI model is used to organize the information and summarize it in a form that is easy for humans to understand. Specific operations include generating sentences, creating graphs, and formatting lists. The input is the collected information, and the output is the generated documents.

[0193] Step 6:

[0194] The server uses text-to-speech (TTS) technology (e.g., Azure Cognitive Services) to convert the generated material into emotive speech. Specifically, the TTS engine takes text as input and generates speech data containing emotional expressions. The input is the generated text material, and the output is speech data.

[0195] Step 7:

[0196] The server sends the generated audio data to the device, which receives it and plays it through the speaker. Specifically, the device downloads the audio file and plays it through the audio output device. The input is audio data, and the output is audio data playback for the user.

[0197] Example: Prompt sentence

[0198] For example, a user may input a prompt such as "Tell me about a recommended restaurant near the next rest stop." The entire system operates based on this prompt.

[0199] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0200] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system also includes a function for adjusting the content of responses and the tone of the voice output by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0201] System Configuration

[0202] This system is realized through the cooperation of a user terminal, a server, and an emotion engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[0203] Program operation overview

[0204] 1. Acquiring voice input

[0205] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[0206] 2. Voice to text conversion

[0207] The device receives the user's voice input and converts it into text data using speech recognition software.

[0208] 3. Sending text data to the server

[0209] The terminal sends the converted text data to the server, which then sends the data using the appropriate communication protocol.

[0210] 4. Emotion recognition

[0211] The server utilizes an emotion engine to recognize emotions contained in the user's voice input, which allows for real-time analysis of the user's emotional state when asking a question.

[0212] 5. Information gathering and document preparation

[0213] The server uses generative AI to gather information based on the received text data and the user's perceived emotions. For example, it retrieves the latest information on the global economy in 2023 from an external database. The user's emotional state is reflected in the tone and content of the information.

[0214] 6. Emotional voice conversion

[0215] The server uses text-to-speech (TTS) technology to convert the created material into expressive speech, with intonation and speed that reflects the user's emotions.

[0216] 7. Sending audio data to the device and playing it back

[0217] The server transmits the generated voice data to the terminal, which then plays back the received voice data and conveys the content of the material to the user in an emotional way.

[0218] Specific examples

[0219] Example 1: Requesting an economic report

[0220] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[0221] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0222] 3. The device sends the text data to the server.

[0223] 4. The server uses an emotion engine to recognize emotions such as impatience and excitement from the user's voice.

[0224] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[0225] 6. The server generates the following statement in an emotive voice: "The global economy in 2023 will be extremely unstable..." but delivers it smoothly and calmly.

[0226] 7. The server sends the audio data to the terminal, which plays it and reports it to the user.

[0227] This system allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, thereby saving time and effort and enabling the transmission of high-quality information.

[0228] The processing flow will be explained below.

[0229] Step 1:

[0230] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[0231] Step 2:

[0232] The device receives the user's voice input through a microphone and passes the voice data to voice recognition software.

[0233] Step 3:

[0234] The device uses voice recognition software to convert the received voice data into text data, which results in the text data "Research the world economy in 2023 and create a document."

[0235] Step 4:

[0236] The terminal transmits the converted text data to the server using an appropriate communication protocol (e.g., HTTPS).

[0237] Step 5:

[0238] The server receives the text data and simultaneously analyzes the emotions contained in the user's voice input using an emotion engine, recognizing, for example, impatience, excitement, calmness, etc.

[0239] Step 6:

[0240] The server uses generative AI to collect necessary information based on the received text data and the recognized emotions, such as the latest information on the global economy in 2023, from external databases and the internet.

[0241] Step 7:

[0242] The server organizes the collected information and creates reports and materials that reflect the emotional state recognized by the emotion engine. For example, if the user is excited, the server creates materials that prioritize providing detailed and prompt information.

[0243] Step 8:

[0244] The server uses text-to-speech (TTS) technology to generate emotionally rich voice data based on the created materials, and the voice is played back with intonation and speed that corresponds to the user's emotional state.

[0245] Step 9:

[0246] The server then transmits the generated voice data to the terminal, again using a communication protocol.

[0247] Step 10:

[0248] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[0249] This series of processing steps allows users to efficiently and effectively request information research and document creation, and receive the results in an emotionally rich voice. This system significantly reduces time and effort, and ensures high-quality information transmission.

[0250] Example 2

[0251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0252] Conventional voice input systems provide information and create materials without considering the user's emotions, often generating results that do not match the user's intentions or emotions. Furthermore, information retrieval and material creation may not be performed quickly, resulting in a decrease in user satisfaction. Furthermore, the generated materials cannot be converted into emotionally rich speech, which is a problem that prevents the presentation from being fully effective for the user.

[0253] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for recognizing the user's emotion based on the received text data, means for collecting information based on the recognized emotion information and the text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, and means for playing back the voice. This makes it possible to quickly and effectively collect information and create materials while taking the user's emotions into consideration, and to improve the effectiveness of presentations by providing information to the user in emotive voice.

[0254] "Means for receiving voice input from a user" refers to a device or mechanism that receives voice when a user gives instructions or asks a question by voice.

[0255] "Means for converting said voice input into text data" refers to software or algorithms for analyzing received voice input and converting it into a corresponding text format.

[0256] "Means for transmitting converted text data to a server" refers to a device or software for transmitting the text data converted from speech to a server using an appropriate communication protocol.

[0257] "Means for recognizing a user's emotions based on received text data" refers to algorithms or software for analyzing and recognizing a user's emotions or mental state from text data.

[0258] "Means for collecting information based on recognized emotional information and text data" refers to software or algorithms that use the recognized emotional state of a user and text data to collect the required information from external databases or other sources.

[0259] "Means for generating materials from collected information" refers to software or algorithms for organizing collected information into an appropriate format and structuring it into materials.

[0260] "Means for converting generated materials into emotive speech" refers to software or algorithms for generating speech data that makes full use of emotion and intonation based on the content of the created materials.

[0261] The "means for reproducing the audio" refers to a device or mechanism that reproduces the generated audio data on a terminal and conveys it to the user as audio.

[0262] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system includes a function that recognizes the user's emotions and adjusts the response content and tone of the voice output. Specific embodiments of the system are described below.

[0263] System configuration

[0264] This system is realized through the cooperation of a user terminal, a server, and an emotion recognition engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[0265] Hardware and Software Configuration

[0266] User Device: A device that has a microphone for receiving voice input, a processor for speech recognition, and a speaker for audio playback, using speech recognition software (e.g., Google Cloud Speech-to-Text).

[0267] Server: A device equipped with a powerful processor for information gathering, document creation, emotion recognition, and voice conversion. It uses GPT-4 or similar as a generative AI model, and general-purpose emotion analysis software (e.g., IBM Watson (registered trademark)) as an emotion recognition engine. Google Cloud Text-to-Speech is used for voice synthesis.

[0268] Operational Overview

[0269] The operation of this system is as follows.

[0270] Voice input and text conversion: When a user speaks commands or asks questions, the user device captures the speech and converts it into text data using voice recognition software.

[0271] Transmission of text data: The converted text data is transmitted from the user terminal to the server using a secure communication protocol (e.g., HTTPS).

[0272] Emotion recognition: The server inputs the received text data into an emotion recognition engine to recognize the user's emotions, for example, determining whether the user is anxious or excited.

[0273] Information collection and document creation: The server uses a generative AI model to collect information and create documents based on the recognized emotion information and text data.

[0274] Emotionally rich speech generation: The created materials are converted into speech data that reflects the user's emotions. The speech synthesis software adjusts intonation and speed according to the emotion.

[0275] Transmission and playback of voice data: The generated voice data is transmitted from the server to the user terminal and played back on the user terminal.

[0276] Specific examples

[0277] Example 1: Requesting an economic report

[0278] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[0279] 2. The user device converts this voice instruction into the text "Research the world economy in 2023 and create materials."

[0280] 3. The user terminal sends the text data to the server.

[0281] 4. The server uses an emotion recognition engine to recognize emotions such as impatience and excitement from the user's voice.

[0282] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[0283] 6. The server generates the following statement in an emotive voice, with a calm and smooth intonation: "The global economy in 2023 will be extremely unstable..."

[0284] 7. The server sends the audio data to the terminal, and the user terminal plays it and reports it to the user.

[0285] This allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, saving time and effort while enabling high-quality information transmission.

[0286] Example prompt statement:

[0287] Generate the output speech when the user requests, "Find the latest statistics on climate change and create a report," but in a calm tone.

[0288] By using such prompts, the generative AI model can create and voice materials that meet the user's needs.

[0289] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0290] Step 1:

[0291] The user speaks instructions and questions into the terminal. This voice input is the starting point of the system, and specifically, the input is something like "Research the world economy in 2023 and create materials." This voice data becomes the raw data that will be processed in the next step.

[0292] Step 2:

[0293] When the device receives a user's voice input, it uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. The input here is voice data, and the output is the corresponding text data. The voice waveform data is analyzed, and a language model is applied based on the phonetic patterns to convert it into a string of characters.

[0294] Step 3:

[0295] The terminal sends the converted text data to the server using a secure communication protocol such as HTTPS, and the converted text data is sent as input to the server. During this transmission process, the data is encrypted and divided into packets.

[0296] Step 4:

[0297] The server analyzes the received text data and inputs it into an emotion recognition engine. The emotion recognition engine (e.g., IBM Watson) extracts the user's emotion from the input text. At this stage, the input is text data, and the output is emotion data such as "anxiety" or "excitement." Natural language processing technology is used to infer emotions from context and vocabulary.

[0298] Step 5:

[0299] The server uses a generative AI model (e.g., OpenAI's GPT-4) to collect information and create materials based on the recognized emotional information and text data. The input here is emotional information and text data, and the output is specific materials (in text format). Information is collected using an external database API, and the generative AI model summarizes and analyzes the collected data.

[0300] Step 6:

[0301] The server converts the created material into emotionally rich audio data using text-to-speech (TTS) technology (e.g., Google Cloud Text-to-Speech). The input is the text data of the material, and the output is audio data. Processing is performed to adjust intonation and speed based on the emotional information.

[0302] Step 7:

[0303] The server sends the generated voice data to the user's device. The device plays the received voice data and conveys the content to the user. The input here is voice data, and the output is voice playback to the user. An emotionally rich voice is played through the device's speaker.

[0304] Through these processing steps, users can efficiently gather information and receive emotionally appropriate presentations, thereby saving time and effort and enabling high-quality information transmission.

[0305] (Application example 2)

[0306] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0307] In modern factories, workers need to report problems in real time and receive fast, accurate solutions. However, with current systems, it can take a long time from the time a problem is reported to the time a solution is provided, which can lead to work stoppages and delays. Additionally, solution presentations often lack emotional explanations, making it difficult for workers to fully understand the content and take prompt action.

[0308] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0309] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, means for adjusting the generated voice to match the user's emotional state, and means for playing back the voice, thereby enabling workers to report problems in real time, provide problem-solving methods by referring to a past database, and explain the solutions in emotive voice.

[0310] A "means for receiving voice input from a user" is a device that captures commands or questions spoken by a user through a voice input device such as a microphone.

[0311] The "means for converting the voice input into text data" refers to software or a system that uses voice recognition technology to convert the user's voice input into text information.

[0312] The "means for transmitting the converted text data to the server" refers to a protocol and device for transferring the text data to the server using a communication network such as the Internet.

[0313] The "means for collecting information based on received text data" is a system that collects related information from internal and external databases and information sources according to the content of the text data.

[0314] "Means for generating collected information as materials" refers to a system that organizes collected information and creates reports and documents in a format that is easy for users to understand.

[0315] The "means for converting generated materials into emotive speech" is a system that converts the created text material into speech using text-to-speech technology and adds intonation and speed that match the user's emotions.

[0316] "Means for adjusting the generated voice to match the emotional state of the user" refers to a technology that analyzes the user's emotions and adjusts the tone, speed, and intonation of the voice as needed.

[0317] The "means for reproducing the audio" is a device that allows the user to listen to the generated audio data through an audio output device such as a speaker.

[0318] The system of the present invention aims to manage production processes and support problem solving in factories. The system includes a user terminal, a server, an emotion engine, and a voice output device.

[0319] System Configuration

[0320] 1. User Device

[0321] This includes microphones installed on smartphones and factory robots as voice input devices.

[0322] As a speech recognition software, it uses Google Cloud Speech-to-Text API to convert user voice input into text data.

[0323] 2. Server

[0324] The text data is received and the user's emotional state is analyzed using IBM Watson Tone Analyzer.

[0325] OpenAI's GPT-4 is used as the generative AI model to collect relevant information and generate materials based on the received text data.

[0326] Based on the collected information, the material is converted into emotive speech using Amazon Polly.

[0327] The audio data is sent to the terminal and a playback instruction is issued.

[0328] 3. Emotion Engine

[0329] The server uses IBM Watson Tone Analyzer to recognize the user's emotions contained in the voice input and adjust the tone of the generated voice.

[0330] 4. Audio Output Device

[0331] These include speakers built into smartphones and factory robots, which play back audio data to users and provide them with work instructions and problem-solving methods.

[0332] Specific examples

[0333] If a factory worker notices a problem on the production line, they can use the system to report the problem in real time. For example, the worker can voice-record the following:

[0334] "An error has occurred on production line A. Please investigate the cause."

[0335] This voice input is captured by the microphone of a smartphone or factory robot and converted into text data using the Google Cloud Speech-to-Text API. This text data is then sent to a server via the internet, where the user's emotional state is analyzed using IBM Watson Tone Analyzer. OpenAI's GPT-4 is then used to collect relevant information from a database and generate a document proposing solutions to the problem.

[0336] The generated materials are converted into speech using Amazon Polly, with the tone adjusted to match the worker's emotion. The speech data is then sent to a terminal and played back through the worker's smartphone or the factory robot's speaker.

[0337] Prompt Sentence Examples

[0338] An error has occurred on production line A. Please prepare a report in a matter-of-fact manner, including any similar errors that have occurred in the past and solutions to those errors.

[0339] In this manner, the system of the present invention provides a means for rapid and effective problem resolution within the factory.

[0340] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0341] Step 1:

[0342] Acquiring voice input

[0343] The user issues instructions or asks questions by voice into the microphone of a smartphone or factory robot. For example, they might type, "An error has occurred on production line A. Please investigate the cause."

[0344] Input: User's voice commands

[0345] Output: Audio data collected by the microphone

[0346] Step 2:

[0347] Voice to text conversion

[0348] The device receives the collected voice data and converts it into text data using the Google Cloud Speech-to-Text API.

[0349] Input: Audio data

[0350] Output: Converted text data

[0351] Step 3:

[0352] Sending text data to the server

[0353] The terminal sends the converted text data to the server using the HTTPS protocol.

[0354] Input: Text data

[0355] Output: Text data sent to the server

[0356] Step 4:

[0357] emotion recognition

[0358] The server inputs the received text data into the IBM Watson Tone Analyzer to analyze the user's emotional state.

[0359] Input: Text data

[0360] Output: Parsed emotion information

[0361] Step 5:

[0362] Information gathering and document creation

[0363] The server uses OpenAI's GPT-4 to collect relevant information from a database using the analyzed emotional information and text data, and generates a document containing specific solutions, using the prompt, "An error has occurred on production line A. Please prepare a report in a calm tone, including past similar errors and solutions."

[0364] Input: Analyzed emotion information, text data

[0365] Output: Generated material

[0366] Step 6:

[0367] Emotional voice conversion

[0368] The server converts the generated material into audio data using Amazon Polly, adjusting the intonation and speed to match the user's emotional state.

[0369] Input: Generated material

[0370] Output: Emotionally rich audio data

[0371] Step 7:

[0372] Sending audio data to the device

[0373] The server sends the generated audio data to the terminal using the HTTPS protocol.

[0374] Input: Emotionally rich speech data

[0375] Output: Audio data sent to the device

[0376] Step 8:

[0377] Playing audio data

[0378] The terminal reproduces the received audio data through a speaker to convey information to the user.

[0379] Input: Transmitted audio data

[0380] Output: The audio data played to the user

[0381] This process allows users to report problems in the factory in real time and receive fast, accurate solutions.

[0382] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0384] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0385] [Second embodiment]

[0386] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0387] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0388] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0389] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0390] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0392] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0393] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0394] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0395] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0396] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0397] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0398] The present invention is a system that allows a user to request information research or the creation of materials through voice input, and then presents the created materials in an expressive voice. Specific embodiments of the present invention will be described below.

[0399] System Configuration

[0400] This system is realized by the cooperation of the user terminal and the server. The user terminal mainly handles voice input and processing, while the server is responsible for collecting information and creating materials.

[0401] Program operation overview

[0402] 1. Acquiring voice input

[0403] The user issues voice instructions to the device, such as "Research the world economy in 2023 and create a document."

[0404] 2. Voice to text conversion

[0405] The device receives voice input and converts it to text using speech recognition software, which processes the voice data as text.

[0406] 3. Sending text data to the server

[0407] The device then sends the converted text data to the server, using an appropriate API or similar.

[0408] 4. Information gathering and document preparation

[0409] The server uses generative AI based on the received text data to collect information, for example, by retrieving necessary information from external databases or the internet and creating documents based on that information.

[0410] 5. Conversion to emotionally rich speech

[0411] The server converts the generated material into emotive speech using text-to-speech (TTS) technology, which translates the material into natural intonation.

[0412] 6. Sending audio data to the device and playing it back

[0413] After generating the audio data, the server transmits it to the terminal, which then plays the received audio data in a format that presents it to the user.

[0414] Specific examples

[0415] Example 1: Requesting an economic report

[0416] 1. The user says, "Research the world economy in 2023 and create a document."

[0417] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0418] 3. The device sends this text data to the server.

[0419] 4. The server uses generative AI to collect the latest information on the global economy in 2023 and create a report.

[0420] 5. The server generates an emotive voice based on this report, for example, "The global economy in 2023 will be very unstable..."

[0421] 6. The server sends the audio file to the terminal, which plays it and reports it to the user.

[0422] In this way, by utilizing this system, users can easily request information research and document preparation, and receive the results in high-quality audio, enabling efficient and effective information gathering, document preparation, and presentations.

[0423] The processing flow will be explained below.

[0424] Step 1:

[0425] The user speaks to the device, for example, "Research the world economy in 2023 and create a document."

[0426] Step 2:

[0427] The device receives the user's voice input through a microphone and then uses speech recognition software to convert the received speech into text data.

[0428] Step 3:

[0429] The device sends the converted text data to the server, using an appropriate API to send the text data to the server over the network.

[0430] Step 4:

[0431] The server receives the text data and passes it to the generation AI, which then collects the necessary information based on the text data and creates materials. For example, it may collect information about the world economy in 2023 from external databases or the internet.

[0432] Step 5:

[0433] The server creates documents based on the collected information, which are organized in text format and clearly describe the content to be provided to users.

[0434] Step 6:

[0435] The server converts the created materials into emotionally rich speech, using text-to-speech (TTS) technology to generate speech data incorporating natural intonation and emotional expressions.

[0436] Step 7:

[0437] The server then sends the generated voice data to the device, which is also sent via the network via the API.

[0438] Step 8:

[0439] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[0440] This series of processing steps allows users to efficiently request information research and document creation, and receive the results in an expressive voice. This system significantly reduces users' time and effort, and enables high-quality presentations.

[0441] Example 1

[0442] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0443] In recent years, there has been a demand for more efficient information research and document creation, but the time and effort required for these tasks remains a challenge. In particular, there is a demand for systems that can efficiently and automatically perform a series of tasks, including collecting information using voice input, generating documents, and then presenting those documents with expressive voice. However, conventional systems require some of these processes to be performed manually, which can result in reduced efficiency.

[0444] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0445] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text information, and means for transmitting the converted text information to a data processing device, thereby enabling efficient information gathering and document generation based on the voice input, and the results to be presented in an expressive voice.

[0446] "User" refers to a person who requests information research or document creation through voice input.

[0447] "Voice input" refers to the voice that a user speaks into a terminal, and is a means for making requests for information research or document creation by voice.

[0448] "Text information" refers to text data converted from voice input using voice recognition technology.

[0449] "Data processing device" refers to a computer system that receives text information and performs processes such as information collection, document creation, and audio data generation.

[0450] "Data" refers to information such as documents, numbers, statistics, and images obtained as a result of information gathering, and serves as the basis for creating materials.

[0451] "Document" refers to a material generated based on information collected by a data processing device and expressed in text format.

[0452] "Audio data" refers to data in which a document has been converted into expressive audio, and refers to an audio file that is output by a playback device.

[0453] A "playback device" refers to equipment or software that allows users to listen to audio data, and includes, for example, speakers and media players.

[0454] This invention is a system in which a user can request information research or document creation through voice input, and the created document is then presented in an expressive voice. This system is realized mainly through the cooperation of a user terminal and a server.

[0455] System Configuration

[0456] This system consists of the following components:

[0457] User device: A device that receives voice input and converts it into text, using good voice recognition software (e.g., Google Speech-to-Text API).

[0458] Server: A device that collects information, generates documents, and generates emotive voice data based on text information. Here, it uses generative AI models (e.g., OpenAI GPT-4) and speech synthesis technology (e.g., Amazon Polly).

[0459] Communication method: API communication (e.g., HTTP POST) is used to send and receive data between the user terminal and the server.

[0460] Processing flow

[0461] 1. Acquisition of voice input: The user speaks into the device to request specific research or document creation. The device uses the built-in microphone to capture the voice.

[0462] 2. Speech-to-text conversion: The device uses the Google Speech-to-Text API to convert speech into text, turning the speech data into processable text.

[0463] 3. Sending text data to the server: The device sends the converted text data to the server via an HTTP POST request, which includes the destination URL and authentication information.

[0464] 4. Information collection and document creation: The server analyzes the received text data and uses a generative AI model (GPT-4) to collect the necessary information and create documents. The necessary information is collected from external databases and the Internet.

[0465] 5. Conversion to emotive speech: The server sends the generated material to Amazon Polly, which converts it into natural, emotive speech data.

[0466] 6. Sending and playing back audio data to the terminal: The server sends the generated audio file to the terminal, and the terminal plays back the audio data in a way that presents it to the user.

[0467] Specific examples

[0468] Example 1: Requesting an economic report

[0469] 1. The user says, "Research the world economy in 2023 and create a document."

[0470] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0471] 3. The device sends this text data to the server.

[0472] 4. The server uses a generative AI model (GPT-4) to collect the latest information on the global economy in 2023 and create a report.

[0473] 5. The server generates emotive audio based on this report, creating a presentation audio with the following content, for example, "The global economy in 2023 will be extremely unstable..."

[0474] 6. The server sends the audio data to the terminal, which plays it and reports it to the user.

[0475] Prompt Sentence Examples

[0476] "Research and write a detailed report on the global economy in 2023. Then, turn that report into an audio presentation."

[0477] "Gather information about the latest impacts of climate change, create materials and give an emotive presentation."

[0478] By utilizing this system, users can efficiently conduct information research, create documents, and make presentations.

[0479] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0480] Step 1:

[0481] The user speaks to the device to request specific research or document creation. The input is the user's voice, such as "Research the world economy in 2023 and create a document." The device picks up this voice using its built-in microphone.

[0482] Step 2:

[0483] The device sends the acquired voice data to the Google Speech-to-Text API and converts the voice into text information. The input is the voice data acquired in step 1, and text information is obtained by sending this voice data to the API. The output is text information in text format.

[0484] Specific behavior:

[0485] 1. Obtain the audio file recorded by the device.

[0486] 2. Send the acquired audio file to the Google Speech-to-Text API.

[0487] 3. Save the text information returned as an API response.

[0488] Step 3:

[0489] The terminal sends the converted text information to the server via an HTTP POST request. The input is the text information obtained in step 2, and an HTTP request is created to send this information to the server. The output is a response indicating that the request was successful.

[0490] Specific behavior:

[0491] 1. Convert the text information into JSON format and set it as the payload of the HTTP POST request.

[0492] 2. Set authentication information and destination URL in the request header.

[0493] 3. Send an HTTP request to the server and receive a response.

[0494] Step 4:

[0495] The server analyzes the received text information, collects the necessary information using a generative AI model (e.g., GPT-4), and generates the document. The input is the text information sent in step 3, and based on this information, it sends a prompt to the AI ​​model. The output is the generated document.

[0496] Specific behavior:

[0497] 1. The server analyzes the received text information.

[0498] 2. Generate a prompt for GPT-4 to begin collecting information.

[0499] 3. Create materials (documents) based on the returned information.

[0500] Step 5:

[0501] The server sends the created material to Amazon Polly, which converts it into natural, emotive speech data. The input is the material generated in step 4, which is sent to a text-to-speech API to be converted into speech. The output is an audio file.

[0502] Specific behavior:

[0503] 1. Save the generated material in text format.

[0504] 2. Send this text data to the Amazon Polly API.

[0505] 3. Save the audio file (e.g., MP3 format) returned as an API response.

[0506] Step 6:

[0507] The server sends the generated audio file to the terminal. The terminal receives this audio data and plays it in a form that is presented to the user. The input is the audio file generated in step 5, which is sent from the server to the terminal. The output is the playback of the audio file.

[0508] Specific behavior:

[0509] 1. The server sends the audio file in an HTTP response.

[0510] 2. The device saves the received audio file locally.

[0511] 3. Play the saved audio file in a media player and let the user listen to it.

[0512] Through this series of processes, the user can automatically conduct research based on the content specified by voice, create materials, and receive the content in an expressive voice.

[0513] (Application example 1)

[0514] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0515] In conventional self-driving vehicles, when a user requests information by voice, it is difficult to search for information or generate materials in real time. It is also difficult to present the acquired information in an expressive voice. As a result, users often experience inconvenience because they are unable to immediately obtain useful information about their destination or current situation. The present invention aims to solve these problems and provide a system that enables real-time, expressive voice presentations in self-driving vehicles.

[0516] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0517] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into expressive voice, means for use in an autonomous vehicle, means for providing information and generating materials in real time, means for generating materials using a generative AI model, and means for transmitting and playing voice to a terminal in the autonomous vehicle. This enables a user to obtain necessary information in real time in an autonomous vehicle and receive the information in expressive voice.

[0518] "Means for receiving voice input" refers to a device or software that recognizes the voice uttered by the user and inputs the voice data into the system.

[0519] The "means for converting speech input into text data" refers to a device or software that analyzes received speech data and outputs the resulting text data.

[0520] The "means for transmitting the converted text data to the server" refers to a communication device or software for sending the text data to the server via a network.

[0521] "Information gathering means" refers to a device or software that searches and retrieves the required information from a server or other database.

[0522] "Means for generating materials" refers to devices or software that process collected information and compile it into a format that is easy for users to use.

[0523] The "means for converting into emotionally rich speech" is a device or software that generates speech with natural intonation including emotional expressions based on the generated text data.

[0524] The "means for reproducing audio" is a device such as a speaker for reproducing the generated audio data as actual sound.

[0525] "In-vehicle means" means any device or software that is installed and operates within an automated vehicle.

[0526] "Means for providing information and generating materials in real time" refers to a device or software for instantly collecting information and generating materials in response to user instructions.

[0527] A "means for generating materials using a generative AI model" is a device or software that uses advanced artificial intelligence to collect information and generate materials.

[0528] The "means for transmitting audio to a terminal and playing it back" refers to a device or software that transmits audio data generated by a server to a terminal and plays back the audio on the terminal.

[0529] The present invention is a system that provides real-time information within an autonomous vehicle, generates materials, and reports them to the user in an expressive voice. The system receives voice input, converts the content into text data, and sends it to a server. The server collects information based on the received text data and generates materials using a generative AI model. The generated materials are converted into expressive voice, and the voice data is sent to a terminal. Finally, the terminal plays the received voice data for the user.

[0530] About program processing

[0531] 1. Acquiring voice input:

[0532] The device inside the car captures the user's voice. This uses voice recognition software (e.g., Google Cloud Speech-to-Text) to capture the user's voice as digital data.

[0533] 2. Voice to text conversion:

[0534] The captured voice data is analyzed and converted into corresponding text data. At this stage, speech recognition software converts speech into text.

[0535] 3. Sending text data:

[0536] The converted text data is sent to the server using the vehicle's communication module, where a RESTful API is used as the communication method.

[0537] 4. Information gathering and data generation:

[0538] The server uses a generative AI model (such as OpenAI's GPT-4) to collect information based on the received text data. Specifically, it searches external databases and information on the internet to collect and analyze the necessary data.

[0539] Based on the collected information, materials are generated. A generative AI model is used to generate these materials.

[0540] 5. Emotional voice conversion:

[0541] The material is converted into emotive audio using text-to-speech (TTS) technology, such as the TTS capabilities of Azure Cognitive Services.

[0542] 6. Playing audio data:

[0543] The server transmits the generated voice data to an in-vehicle terminal, which then plays the voice data through a speaker.

[0544] Specific examples

[0545] For example, suppose a user speaks to an in-car device, saying, "Tell me about recommended restaurants near the next rest stop." This voice input is converted to text using Google Cloud Speech-to-Text and sent to the server. The server uses OpenAI's GPT-4 to collect information about recommended restaurants near the next rest stop from the internet. The collected information is compiled into a document, and the content is converted into emotive speech using Azure Cognitive Services' TTS technology. Finally, the speech is sent to the in-car device and played over the speaker. An example of a prompt sentence is, "Tell me about recommended restaurants near the next rest stop." This system allows users to obtain the information they need in real time and receive it in emotive speech.

[0546] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0547] Step 1:

[0548] The user inputs voice into the terminal inside the autonomous vehicle. Specifically, they speak through the microphone, saying, "Tell me about a recommended restaurant near the next rest stop." This voice data is captured by the terminal. The input is the user's voice, and the output is analog voice data.

[0549] Step 2:

[0550] The device receives voice input and converts the voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Specifically, the voice recognition engine analyzes the sound waveform and generates a corresponding string of characters. The input is voice data, and the output is text data.

[0551] Step 3:

[0552] The device sends the content converted into text data to the server. A RESTful API is used as the communication method, and the text data is sent as an HTTP POST request. Specifically, the device sends the generated text data to the server in JSON format. The input is text data, and the output is data sent to the server.

[0553] Step 4:

[0554] The server uses a generative AI model (such as OpenAI's GPT-4) to collect the necessary information based on the received text data. Specifically, it searches for and retrieves relevant information from the internet and internal databases. The input is text data, and the output is the collected information.

[0555] Step 5:

[0556] The server generates documents based on the collected information. Here too, a generative AI model is used to organize the information and summarize it in a form that is easy for humans to understand. Specific operations include generating sentences, creating graphs, and formatting lists. The input is the collected information, and the output is the generated documents.

[0557] Step 6:

[0558] The server uses text-to-speech (TTS) technology (e.g., Azure Cognitive Services) to convert the generated material into emotive speech. Specifically, the TTS engine takes text as input and generates speech data containing emotional expressions. The input is the generated text material, and the output is speech data.

[0559] Step 7:

[0560] The server sends the generated audio data to the device, which receives it and plays it through the speaker. Specifically, the device downloads the audio file and plays it through the audio output device. The input is audio data, and the output is audio data playback for the user.

[0561] Example: Prompt sentence

[0562] For example, a user may input a prompt such as "Tell me about a recommended restaurant near the next rest stop." The entire system operates based on this prompt.

[0563] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0564] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system also includes a function for adjusting the content of responses and the tone of the voice output by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0565] System Configuration

[0566] This system is realized through the cooperation of a user terminal, a server, and an emotion engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[0567] Program operation overview

[0568] 1. Acquiring voice input

[0569] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[0570] 2. Voice to text conversion

[0571] The device receives the user's voice input and converts it into text data using speech recognition software.

[0572] 3. Sending text data to the server

[0573] The terminal sends the converted text data to the server, which then sends the data using the appropriate communication protocol.

[0574] 4. Emotion recognition

[0575] The server utilizes an emotion engine to recognize emotions contained in the user's voice input, which allows for real-time analysis of the user's emotional state when asking a question.

[0576] 5. Information gathering and document preparation

[0577] The server uses generative AI to gather information based on the received text data and the user's perceived emotions. For example, it retrieves the latest information on the global economy in 2023 from an external database. The user's emotional state is reflected in the tone and content of the information.

[0578] 6. Emotional voice conversion

[0579] The server uses text-to-speech (TTS) technology to convert the created material into expressive speech, with intonation and speed that reflects the user's emotions.

[0580] 7. Sending audio data to the device and playing it back

[0581] The server transmits the generated voice data to the terminal, which then plays back the received voice data and conveys the content of the material to the user in an emotional way.

[0582] Specific examples

[0583] Example 1: Requesting an economic report

[0584] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[0585] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0586] 3. The device sends the text data to the server.

[0587] 4. The server uses an emotion engine to recognize emotions such as impatience and excitement from the user's voice.

[0588] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[0589] 6. The server generates the following statement in an emotive voice: "The global economy in 2023 will be extremely unstable..." but delivers it smoothly and calmly.

[0590] 7. The server sends the audio data to the terminal, which plays it and reports it to the user.

[0591] This system allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, thereby saving time and effort and enabling the transmission of high-quality information.

[0592] The processing flow will be explained below.

[0593] Step 1:

[0594] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[0595] Step 2:

[0596] The device receives the user's voice input through a microphone and passes the voice data to voice recognition software.

[0597] Step 3:

[0598] The device uses voice recognition software to convert the received voice data into text data, which results in the text data "Research the world economy in 2023 and create a document."

[0599] Step 4:

[0600] The terminal transmits the converted text data to the server using an appropriate communication protocol (e.g., HTTPS).

[0601] Step 5:

[0602] The server receives the text data and simultaneously analyzes the emotions contained in the user's voice input using an emotion engine, recognizing, for example, impatience, excitement, calmness, etc.

[0603] Step 6:

[0604] The server uses generative AI to collect necessary information based on the received text data and the recognized emotions, such as the latest information on the global economy in 2023, from external databases and the internet.

[0605] Step 7:

[0606] The server organizes the collected information and creates reports and materials that reflect the emotional state recognized by the emotion engine. For example, if the user is excited, the server creates materials that prioritize providing detailed and prompt information.

[0607] Step 8:

[0608] The server uses text-to-speech (TTS) technology to generate emotionally rich voice data based on the created materials, and the voice is played back with intonation and speed that corresponds to the user's emotional state.

[0609] Step 9:

[0610] The server then transmits the generated voice data to the terminal, again using a communication protocol.

[0611] Step 10:

[0612] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[0613] This series of processing steps allows users to efficiently and effectively request information research and document creation, and receive the results in an emotionally rich voice. This system significantly reduces time and effort, and ensures high-quality information transmission.

[0614] Example 2

[0615] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0616] Conventional voice input systems provide information and create materials without considering the user's emotions, often generating results that do not match the user's intentions or emotions. Furthermore, information retrieval and material creation may not be performed quickly, resulting in a decrease in user satisfaction. Furthermore, the generated materials cannot be converted into emotionally rich speech, which is a problem that prevents the presentation from being fully effective for the user.

[0617] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for recognizing the user's emotion based on the received text data, means for collecting information based on the recognized emotion information and the text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, and means for playing back the voice. This makes it possible to quickly and effectively collect information and create materials while taking the user's emotions into consideration, and to improve the effectiveness of presentations by providing information to the user in emotive voice.

[0618] "Means for receiving voice input from a user" refers to a device or mechanism that receives voice when a user gives instructions or asks a question by voice.

[0619] "Means for converting said voice input into text data" refers to software or algorithms for analyzing received voice input and converting it into a corresponding text format.

[0620] "Means for transmitting converted text data to a server" refers to a device or software for transmitting the text data converted from speech to a server using an appropriate communication protocol.

[0621] "Means for recognizing a user's emotions based on received text data" refers to algorithms or software for analyzing and recognizing a user's emotions or mental state from text data.

[0622] "Means for collecting information based on recognized emotional information and text data" refers to software or algorithms that use the recognized emotional state of a user and text data to collect the required information from external databases or other sources.

[0623] "Means for generating materials from collected information" refers to software or algorithms for organizing collected information into an appropriate format and structuring it into materials.

[0624] "Means for converting generated materials into emotive speech" refers to software or algorithms for generating speech data that makes full use of emotion and intonation based on the content of the created materials.

[0625] The "means for reproducing the audio" refers to a device or mechanism that reproduces the generated audio data on a terminal and conveys it to the user as audio.

[0626] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system includes a function that recognizes the user's emotions and adjusts the response content and tone of the voice output. Specific embodiments of the system are described below.

[0627] System configuration

[0628] This system is realized through the cooperation of a user terminal, a server, and an emotion recognition engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[0629] Hardware and Software Configuration

[0630] User Device: A device that has a microphone for receiving voice input, a processor for speech recognition, and a speaker for audio playback, using speech recognition software (e.g., Google Cloud Speech-to-Text).

[0631] Server: A device equipped with a powerful processor for information gathering, document creation, emotion recognition, and speech conversion. It uses GPT-4 or similar as a generative AI model, and general-purpose emotion analysis software (e.g., IBM Watson) as an emotion recognition engine. Google Cloud Text-to-Speech is used for speech synthesis.

[0632] Operational Overview

[0633] The operation of this system is as follows.

[0634] Voice input and text conversion: When a user speaks commands or asks questions, the user device captures the speech and converts it into text data using voice recognition software.

[0635] Transmission of text data: The converted text data is transmitted from the user terminal to the server using a secure communication protocol (e.g., HTTPS).

[0636] Emotion recognition: The server inputs the received text data into an emotion recognition engine to recognize the user's emotions, for example, determining whether the user is anxious or excited.

[0637] Information collection and document creation: The server uses a generative AI model to collect information and create documents based on the recognized emotion information and text data.

[0638] Emotionally rich speech generation: The created materials are converted into speech data that reflects the user's emotions. The speech synthesis software adjusts intonation and speed according to the emotion.

[0639] Transmission and playback of voice data: The generated voice data is transmitted from the server to the user terminal and played back on the user terminal.

[0640] Specific examples

[0641] Example 1: Requesting an economic report

[0642] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[0643] 2. The user device converts this voice instruction into the text "Research the world economy in 2023 and create materials."

[0644] 3. The user terminal sends the text data to the server.

[0645] 4. The server uses an emotion recognition engine to recognize emotions such as impatience and excitement from the user's voice.

[0646] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[0647] 6. The server generates the following statement in an emotive voice, with a calm and smooth intonation: "The global economy in 2023 will be extremely unstable..."

[0648] 7. The server sends the audio data to the terminal, and the user terminal plays it and reports it to the user.

[0649] This allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, saving time and effort while enabling high-quality information transmission.

[0650] Example prompt statement:

[0651] Generate the output speech when the user requests, "Find the latest statistics on climate change and create a report," but in a calm tone.

[0652] By using such prompts, the generative AI model can create and voice materials that meet the user's needs.

[0653] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0654] Step 1:

[0655] The user speaks instructions and questions into the terminal. This voice input is the starting point of the system, and specifically, the input is something like "Research the world economy in 2023 and create materials." This voice data becomes the raw data that will be processed in the next step.

[0656] Step 2:

[0657] When the device receives a user's voice input, it uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. The input here is voice data, and the output is the corresponding text data. The voice waveform data is analyzed, and a language model is applied based on the phonetic patterns to convert it into a string of characters.

[0658] Step 3:

[0659] The terminal sends the converted text data to the server using a secure communication protocol such as HTTPS, and the converted text data is sent as input to the server. During this transmission process, the data is encrypted and divided into packets.

[0660] Step 4:

[0661] The server analyzes the received text data and inputs it into an emotion recognition engine. The emotion recognition engine (e.g., IBM Watson) extracts the user's emotion from the input text. At this stage, the input is text data, and the output is emotion data such as "anxiety" or "excitement." Natural language processing technology is used to infer emotions from context and vocabulary.

[0662] Step 5:

[0663] The server uses a generative AI model (e.g., OpenAI's GPT-4) to collect information and create materials based on the recognized emotional information and text data. The input here is emotional information and text data, and the output is specific materials (in text format). Information is collected using an external database API, and the generative AI model summarizes and analyzes the collected data.

[0664] Step 6:

[0665] The server converts the created material into emotionally rich audio data using text-to-speech (TTS) technology (e.g., Google Cloud Text-to-Speech). The input is the text data of the material, and the output is audio data. Processing is performed to adjust intonation and speed based on the emotional information.

[0666] Step 7:

[0667] The server sends the generated voice data to the user's device. The device plays the received voice data and conveys the content to the user. The input here is voice data, and the output is voice playback to the user. An emotionally rich voice is played through the device's speaker.

[0668] Through these processing steps, users can efficiently gather information and receive emotionally appropriate presentations, thereby saving time and effort and enabling high-quality information transmission.

[0669] (Application example 2)

[0670] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] In modern factories, workers need to report problems in real time and receive fast, accurate solutions. However, with current systems, it can take a long time from the time a problem is reported to the time a solution is provided, which can lead to work stoppages and delays. Additionally, solution presentations often lack emotional explanations, making it difficult for workers to fully understand the content and take prompt action.

[0672] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0673] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, means for adjusting the generated voice to match the user's emotional state, and means for playing back the voice, thereby enabling workers to report problems in real time, provide problem-solving methods by referring to a past database, and explain the solutions in emotive voice.

[0674] A "means for receiving voice input from a user" is a device that captures commands or questions spoken by a user through a voice input device such as a microphone.

[0675] The "means for converting the voice input into text data" refers to software or a system that uses voice recognition technology to convert the user's voice input into text information.

[0676] The "means for transmitting the converted text data to the server" refers to a protocol and device for transferring the text data to the server using a communication network such as the Internet.

[0677] The "means for collecting information based on received text data" is a system that collects related information from internal and external databases and information sources according to the content of the text data.

[0678] "Means for generating collected information as materials" refers to a system that organizes collected information and creates reports and documents in a format that is easy for users to understand.

[0679] The "means for converting generated materials into emotive speech" is a system that converts the created text material into speech using text-to-speech technology and adds intonation and speed that match the user's emotions.

[0680] "Means for adjusting the generated voice to match the emotional state of the user" refers to a technology that analyzes the user's emotions and adjusts the tone, speed, and intonation of the voice as needed.

[0681] The "means for reproducing the audio" is a device that allows the user to listen to the generated audio data through an audio output device such as a speaker.

[0682] The system of the present invention aims to manage production processes and support problem solving in factories. The system includes a user terminal, a server, an emotion engine, and a voice output device.

[0683] System Configuration

[0684] 1. User Device

[0685] This includes microphones installed on smartphones and factory robots as voice input devices.

[0686] As a speech recognition software, it uses Google Cloud Speech-to-Text API to convert user voice input into text data.

[0687] 2. Server

[0688] The text data is received and the user's emotional state is analyzed using IBM Watson Tone Analyzer.

[0689] OpenAI's GPT-4 is used as the generative AI model to collect relevant information and generate materials based on the received text data.

[0690] Based on the collected information, the material is converted into emotive speech using Amazon Polly.

[0691] The audio data is sent to the terminal and a playback instruction is issued.

[0692] 3. Emotion Engine

[0693] The server uses IBM Watson Tone Analyzer to recognize the user's emotions contained in the voice input and adjust the tone of the generated voice.

[0694] 4. Audio Output Device

[0695] These include speakers built into smartphones and factory robots, which play back audio data to users and provide them with work instructions and problem-solving methods.

[0696] Specific examples

[0697] If a factory worker notices a problem on the production line, they can use the system to report the problem in real time. For example, the worker can voice-record the following:

[0698] "An error has occurred on production line A. Please investigate the cause."

[0699] This voice input is captured by the microphone of a smartphone or factory robot and converted into text data using the Google Cloud Speech-to-Text API. This text data is then sent to a server via the internet, where the user's emotional state is analyzed using IBM Watson Tone Analyzer. OpenAI's GPT-4 is then used to collect relevant information from a database and generate a document proposing solutions to the problem.

[0700] The generated materials are converted into speech using Amazon Polly, with the tone adjusted to match the worker's emotion. The speech data is then sent to a terminal and played back through the worker's smartphone or the factory robot's speaker.

[0701] Prompt Sentence Examples

[0702] An error has occurred on production line A. Please prepare a report in a matter-of-fact manner, including any similar errors that have occurred in the past and solutions to those errors.

[0703] In this manner, the system of the present invention provides a means for rapid and effective problem resolution within the factory.

[0704] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0705] Step 1:

[0706] Acquiring voice input

[0707] The user issues instructions or asks questions by voice into the microphone of a smartphone or factory robot. For example, they might type, "An error has occurred on production line A. Please investigate the cause."

[0708] Input: User's voice commands

[0709] Output: Audio data collected by the microphone

[0710] Step 2:

[0711] Voice to text conversion

[0712] The device receives the collected voice data and converts it into text data using the Google Cloud Speech-to-Text API.

[0713] Input: Audio data

[0714] Output: Converted text data

[0715] Step 3:

[0716] Sending text data to the server

[0717] The terminal sends the converted text data to the server using the HTTPS protocol.

[0718] Input: Text data

[0719] Output: Text data sent to the server

[0720] Step 4:

[0721] emotion recognition

[0722] The server inputs the received text data into the IBM Watson Tone Analyzer to analyze the user's emotional state.

[0723] Input: Text data

[0724] Output: Parsed emotion information

[0725] Step 5:

[0726] Information gathering and document creation

[0727] The server uses OpenAI's GPT-4 to collect relevant information from a database using the analyzed emotional information and text data, and generates a document containing specific solutions, using the prompt, "An error has occurred on production line A. Please prepare a report in a calm tone, including past similar errors and solutions."

[0728] Input: Analyzed emotion information, text data

[0729] Output: Generated material

[0730] Step 6:

[0731] Emotional voice conversion

[0732] The server converts the generated material into audio data using Amazon Polly, adjusting the intonation and speed to match the user's emotional state.

[0733] Input: Generated material

[0734] Output: Emotionally rich audio data

[0735] Step 7:

[0736] Sending audio data to the device

[0737] The server sends the generated audio data to the terminal using the HTTPS protocol.

[0738] Input: Emotionally rich speech data

[0739] Output: Audio data sent to the device

[0740] Step 8:

[0741] Playing audio data

[0742] The terminal reproduces the received audio data through a speaker to convey information to the user.

[0743] Input: Transmitted audio data

[0744] Output: The audio data played to the user

[0745] This process allows users to report problems in the factory in real time and receive fast, accurate solutions.

[0746] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0747] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0748] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0749] [Third embodiment]

[0750] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0751] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0752] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0753] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0754] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0755] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0756] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0757] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0758] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0759] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0760] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0761] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0762] The present invention is a system that allows a user to request information research or the creation of materials through voice input, and then presents the created materials in an expressive voice. Specific embodiments of the present invention will be described below.

[0763] System Configuration

[0764] This system is realized by the cooperation of the user terminal and the server. The user terminal mainly handles voice input and processing, while the server is responsible for collecting information and creating materials.

[0765] Program operation overview

[0766] 1. Acquiring voice input

[0767] The user issues voice instructions to the device, such as "Research the world economy in 2023 and create a document."

[0768] 2. Voice to text conversion

[0769] The device receives voice input and converts it to text using speech recognition software, which processes the voice data as text.

[0770] 3. Sending text data to the server

[0771] The device then sends the converted text data to the server, using an appropriate API or similar.

[0772] 4. Information gathering and document preparation

[0773] The server uses generative AI based on the received text data to collect information, for example, by retrieving necessary information from external databases or the internet and creating documents based on that information.

[0774] 5. Conversion to emotionally rich speech

[0775] The server converts the generated material into emotive speech using text-to-speech (TTS) technology, which translates the material into natural intonation.

[0776] 6. Sending audio data to the device and playing it back

[0777] After generating the audio data, the server transmits it to the terminal, which then plays the received audio data in a format that presents it to the user.

[0778] Specific examples

[0779] Example 1: Requesting an economic report

[0780] 1. The user says, "Research the world economy in 2023 and create a document."

[0781] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0782] 3. The device sends this text data to the server.

[0783] 4. The server uses generative AI to collect the latest information on the global economy in 2023 and create a report.

[0784] 5. The server generates an emotive voice based on this report, for example, "The global economy in 2023 will be very unstable..."

[0785] 6. The server sends the audio file to the terminal, which plays it and reports it to the user.

[0786] In this way, by utilizing this system, users can easily request information research and document preparation, and receive the results in high-quality audio, enabling efficient and effective information gathering, document preparation, and presentations.

[0787] The processing flow will be explained below.

[0788] Step 1:

[0789] The user speaks to the device, for example, "Research the world economy in 2023 and create a document."

[0790] Step 2:

[0791] The device receives the user's voice input through a microphone and then uses speech recognition software to convert the received speech into text data.

[0792] Step 3:

[0793] The device sends the converted text data to the server, using an appropriate API to send the text data to the server over the network.

[0794] Step 4:

[0795] The server receives the text data and passes it to the generation AI, which then collects the necessary information based on the text data and creates materials. For example, it may collect information about the world economy in 2023 from external databases or the internet.

[0796] Step 5:

[0797] The server creates documents based on the collected information, which are organized in text format and clearly describe the content to be provided to users.

[0798] Step 6:

[0799] The server converts the created materials into emotionally rich speech, using text-to-speech (TTS) technology to generate speech data incorporating natural intonation and emotional expressions.

[0800] Step 7:

[0801] The server then sends the generated voice data to the device, which is also sent via the network via the API.

[0802] Step 8:

[0803] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[0804] This series of processing steps allows users to efficiently request information research and document creation, and receive the results in an expressive voice. This system significantly reduces users' time and effort, and enables high-quality presentations.

[0805] Example 1

[0806] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0807] In recent years, there has been a demand for more efficient information research and document creation, but the time and effort required for these tasks remains a challenge. In particular, there is a demand for systems that can efficiently and automatically perform a series of tasks, including collecting information using voice input, generating documents, and then presenting those documents with expressive voice. However, conventional systems require some of these processes to be performed manually, which can result in reduced efficiency.

[0808] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0809] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text information, and means for transmitting the converted text information to a data processing device, thereby enabling efficient information gathering and document generation based on the voice input, and the results to be presented in an expressive voice.

[0810] "User" refers to a person who requests information research or document creation through voice input.

[0811] "Voice input" refers to the voice that a user speaks into a terminal, and is a means for making requests for information research or document creation by voice.

[0812] "Text information" refers to text data converted from voice input using voice recognition technology.

[0813] "Data processing device" refers to a computer system that receives text information and performs processes such as information collection, document creation, and audio data generation.

[0814] "Data" refers to information such as documents, numbers, statistics, and images obtained as a result of information gathering, and serves as the basis for creating materials.

[0815] "Document" refers to a material generated based on information collected by a data processing device and expressed in text format.

[0816] "Audio data" refers to data in which a document has been converted into expressive audio, and refers to an audio file that is output by a playback device.

[0817] A "playback device" refers to equipment or software that allows users to listen to audio data, and includes, for example, speakers and media players.

[0818] This invention is a system in which a user can request information research or document creation through voice input, and the created document is then presented in an expressive voice. This system is realized mainly through the cooperation of a user terminal and a server.

[0819] System Configuration

[0820] This system consists of the following components:

[0821] User device: A device that receives voice input and converts it into text, using good voice recognition software (e.g., Google Speech-to-Text API).

[0822] Server: A device that collects information, generates documents, and generates emotive voice data based on text information. Here, it uses generative AI models (e.g., OpenAI GPT-4) and speech synthesis technology (e.g., Amazon Polly).

[0823] Communication method: API communication (e.g., HTTP POST) is used to send and receive data between the user terminal and the server.

[0824] Processing flow

[0825] 1. Acquisition of voice input: The user speaks into the device to request specific research or document creation. The device uses the built-in microphone to capture the voice.

[0826] 2. Speech-to-text conversion: The device uses the Google Speech-to-Text API to convert speech into text, turning the speech data into processable text.

[0827] 3. Sending text data to the server: The device sends the converted text data to the server via an HTTP POST request, which includes the destination URL and authentication information.

[0828] 4. Information collection and document creation: The server analyzes the received text data and uses a generative AI model (GPT-4) to collect the necessary information and create documents. The necessary information is collected from external databases and the Internet.

[0829] 5. Conversion to emotive speech: The server sends the generated material to Amazon Polly, which converts it into natural, emotive speech data.

[0830] 6. Sending and playing back audio data to the terminal: The server sends the generated audio file to the terminal, and the terminal plays back the audio data in a way that presents it to the user.

[0831] Specific examples

[0832] Example 1: Requesting an economic report

[0833] 1. The user says, "Research the world economy in 2023 and create a document."

[0834] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0835] 3. The device sends this text data to the server.

[0836] 4. The server uses a generative AI model (GPT-4) to collect the latest information on the global economy in 2023 and create a report.

[0837] 5. The server generates emotive audio based on this report, creating a presentation audio with the following content, for example, "The global economy in 2023 will be extremely unstable..."

[0838] 6. The server sends the audio data to the terminal, which plays it and reports it to the user.

[0839] Prompt Sentence Examples

[0840] "Research and write a detailed report on the global economy in 2023. Then, turn that report into an audio presentation."

[0841] "Gather information about the latest impacts of climate change, create materials and give an emotive presentation."

[0842] By utilizing this system, users can efficiently conduct information research, create documents, and make presentations.

[0843] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0844] Step 1:

[0845] The user speaks to the device to request specific research or document creation. The input is the user's voice, such as "Research the world economy in 2023 and create a document." The device picks up this voice using its built-in microphone.

[0846] Step 2:

[0847] The device sends the acquired voice data to the Google Speech-to-Text API and converts the voice into text information. The input is the voice data acquired in step 1, and text information is obtained by sending this voice data to the API. The output is text information in text format.

[0848] Specific behavior:

[0849] 1. Obtain the audio file recorded by the device.

[0850] 2. Send the acquired audio file to the Google Speech-to-Text API.

[0851] 3. Save the text information returned as an API response.

[0852] Step 3:

[0853] The terminal sends the converted text information to the server via an HTTP POST request. The input is the text information obtained in step 2, and an HTTP request is created to send this information to the server. The output is a response indicating that the request was successful.

[0854] Specific behavior:

[0855] 1. Convert the text information into JSON format and set it as the payload of the HTTP POST request.

[0856] 2. Set authentication information and destination URL in the request header.

[0857] 3. Send an HTTP request to the server and receive a response.

[0858] Step 4:

[0859] The server analyzes the received text information, collects the necessary information using a generative AI model (e.g., GPT-4), and generates the document. The input is the text information sent in step 3, and based on this information, it sends a prompt to the AI ​​model. The output is the generated document.

[0860] Specific behavior:

[0861] 1. The server analyzes the received text information.

[0862] 2. Generate a prompt for GPT-4 to begin collecting information.

[0863] 3. Create materials (documents) based on the returned information.

[0864] Step 5:

[0865] The server sends the created material to Amazon Polly, which converts it into natural, emotive speech data. The input is the material generated in step 4, which is sent to a text-to-speech API to be converted into speech. The output is an audio file.

[0866] Specific behavior:

[0867] 1. Save the generated material in text format.

[0868] 2. Send this text data to the Amazon Polly API.

[0869] 3. Save the audio file (e.g., MP3 format) returned as an API response.

[0870] Step 6:

[0871] The server sends the generated audio file to the terminal. The terminal receives this audio data and plays it in a form that is presented to the user. The input is the audio file generated in step 5, which is sent from the server to the terminal. The output is the playback of the audio file.

[0872] Specific behavior:

[0873] 1. The server sends the audio file in an HTTP response.

[0874] 2. The device saves the received audio file locally.

[0875] 3. Play the saved audio file in a media player and let the user listen to it.

[0876] Through this series of processes, the user can automatically conduct research based on the content specified by voice, create materials, and receive the content in an expressive voice.

[0877] (Application example 1)

[0878] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0879] In conventional self-driving vehicles, when a user requests information by voice, it is difficult to search for information or generate materials in real time. It is also difficult to present the acquired information in an expressive voice. As a result, users often experience inconvenience because they are unable to immediately obtain useful information about their destination or current situation. The present invention aims to solve these problems and provide a system that enables real-time, expressive voice presentations in self-driving vehicles.

[0880] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0881] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into expressive voice, means for use in an autonomous vehicle, means for providing information and generating materials in real time, means for generating materials using a generative AI model, and means for transmitting and playing voice to a terminal in the autonomous vehicle. This enables a user to obtain necessary information in real time in an autonomous vehicle and receive the information in expressive voice.

[0882] "Means for receiving voice input" refers to a device or software that recognizes the voice uttered by the user and inputs the voice data into the system.

[0883] The "means for converting speech input into text data" refers to a device or software that analyzes received speech data and outputs the resulting text data.

[0884] The "means for transmitting the converted text data to the server" refers to a communication device or software for sending the text data to the server via a network.

[0885] "Information gathering means" refers to a device or software that searches and retrieves the required information from a server or other database.

[0886] "Means for generating materials" refers to devices or software that process collected information and compile it into a format that is easy for users to use.

[0887] The "means for converting into emotionally rich speech" is a device or software that generates speech with natural intonation including emotional expressions based on the generated text data.

[0888] The "means for reproducing audio" is a device such as a speaker for reproducing the generated audio data as actual sound.

[0889] "In-vehicle means" means any device or software that is installed and operates within an automated vehicle.

[0890] "Means for providing information and generating materials in real time" refers to a device or software for instantly collecting information and generating materials in response to user instructions.

[0891] A "means for generating materials using a generative AI model" is a device or software that uses advanced artificial intelligence to collect information and generate materials.

[0892] The "means for transmitting audio to a terminal and playing it back" refers to a device or software that transmits audio data generated by a server to a terminal and plays back the audio on the terminal.

[0893] The present invention is a system that provides real-time information within an autonomous vehicle, generates materials, and reports them to the user in an expressive voice. The system receives voice input, converts the content into text data, and sends it to a server. The server collects information based on the received text data and generates materials using a generative AI model. The generated materials are converted into expressive voice, and the voice data is sent to a terminal. Finally, the terminal plays the received voice data for the user.

[0894] About program processing

[0895] 1. Acquiring voice input:

[0896] The device inside the car captures the user's voice. This uses voice recognition software (e.g., Google Cloud Speech-to-Text) to capture the user's voice as digital data.

[0897] 2. Voice to text conversion:

[0898] The captured voice data is analyzed and converted into corresponding text data. At this stage, speech recognition software converts speech into text.

[0899] 3. Sending text data:

[0900] The converted text data is sent to the server using the vehicle's communication module, where a RESTful API is used as the communication method.

[0901] 4. Information gathering and data generation:

[0902] The server uses a generative AI model (such as OpenAI's GPT-4) to collect information based on the received text data. Specifically, it searches external databases and information on the internet to collect and analyze the necessary data.

[0903] Based on the collected information, materials are generated. A generative AI model is used to generate these materials.

[0904] 5. Emotional voice conversion:

[0905] The material is converted into emotive audio using text-to-speech (TTS) technology, such as the TTS capabilities of Azure Cognitive Services.

[0906] 6. Playing audio data:

[0907] The server transmits the generated voice data to an in-vehicle terminal, which then plays the voice data through a speaker.

[0908] Specific examples

[0909] For example, suppose a user speaks to an in-car device, saying, "Tell me about recommended restaurants near the next rest stop." This voice input is converted to text using Google Cloud Speech-to-Text and sent to the server. The server uses OpenAI's GPT-4 to collect information about recommended restaurants near the next rest stop from the internet. The collected information is compiled into a document, and the content is converted into emotive speech using Azure Cognitive Services' TTS technology. Finally, the speech is sent to the in-car device and played over the speaker. An example of a prompt sentence is, "Tell me about recommended restaurants near the next rest stop." This system allows users to obtain the information they need in real time and receive it in emotive speech.

[0910] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0911] Step 1:

[0912] The user inputs voice into the terminal inside the autonomous vehicle. Specifically, they speak through the microphone, saying, "Tell me about a recommended restaurant near the next rest stop." This voice data is captured by the terminal. The input is the user's voice, and the output is analog voice data.

[0913] Step 2:

[0914] The device receives voice input and converts the voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Specifically, the voice recognition engine analyzes the sound waveform and generates a corresponding string of characters. The input is voice data, and the output is text data.

[0915] Step 3:

[0916] The device sends the content converted into text data to the server. A RESTful API is used as the communication method, and the text data is sent as an HTTP POST request. Specifically, the device sends the generated text data to the server in JSON format. The input is text data, and the output is data sent to the server.

[0917] Step 4:

[0918] The server uses a generative AI model (such as OpenAI's GPT-4) to collect the necessary information based on the received text data. Specifically, it searches for and retrieves relevant information from the internet and internal databases. The input is text data, and the output is the collected information.

[0919] Step 5:

[0920] The server generates documents based on the collected information. Here too, a generative AI model is used to organize the information and summarize it in a form that is easy for humans to understand. Specific operations include generating sentences, creating graphs, and formatting lists. The input is the collected information, and the output is the generated documents.

[0921] Step 6:

[0922] The server uses text-to-speech (TTS) technology (e.g., Azure Cognitive Services) to convert the generated material into emotive speech. Specifically, the TTS engine takes text as input and generates speech data containing emotional expressions. The input is the generated text material, and the output is speech data.

[0923] Step 7:

[0924] The server sends the generated audio data to the device, which receives it and plays it through the speaker. Specifically, the device downloads the audio file and plays it through the audio output device. The input is audio data, and the output is audio data playback for the user.

[0925] Example: Prompt sentence

[0926] For example, a user may input a prompt such as "Tell me about a recommended restaurant near the next rest stop." The entire system operates based on this prompt.

[0927] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0928] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system also includes a function for adjusting the content of responses and the tone of the voice output by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[0929] System Configuration

[0930] This system is realized through the cooperation of a user terminal, a server, and an emotion engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[0931] Program operation overview

[0932] 1. Acquiring voice input

[0933] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[0934] 2. Voice to text conversion

[0935] The device receives the user's voice input and converts it into text data using speech recognition software.

[0936] 3. Sending text data to the server

[0937] The terminal sends the converted text data to the server, which then sends the data using the appropriate communication protocol.

[0938] 4. Emotion recognition

[0939] The server utilizes an emotion engine to recognize emotions contained in the user's voice input, which allows for real-time analysis of the user's emotional state when asking a question.

[0940] 5. Information gathering and document preparation

[0941] The server uses generative AI to gather information based on the received text data and the user's perceived emotions. For example, it retrieves the latest information on the global economy in 2023 from an external database. The user's emotional state is reflected in the tone and content of the information.

[0942] 6. Emotional voice conversion

[0943] The server uses text-to-speech (TTS) technology to convert the created material into expressive speech, with intonation and speed that reflects the user's emotions.

[0944] 7. Sending audio data to the device and playing it back

[0945] The server transmits the generated voice data to the terminal, which then plays back the received voice data and conveys the content of the material to the user in an emotional way.

[0946] Specific examples

[0947] Example 1: Requesting an economic report

[0948] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[0949] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[0950] 3. The device sends the text data to the server.

[0951] 4. The server uses an emotion engine to recognize emotions such as impatience and excitement from the user's voice.

[0952] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[0953] 6. The server generates the following statement in an emotive voice: "The global economy in 2023 will be extremely unstable..." but delivers it smoothly and calmly.

[0954] 7. The server sends the audio data to the terminal, which plays it and reports it to the user.

[0955] This system allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, thereby saving time and effort and enabling the transmission of high-quality information.

[0956] The processing flow will be explained below.

[0957] Step 1:

[0958] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[0959] Step 2:

[0960] The device receives the user's voice input through a microphone and passes the voice data to voice recognition software.

[0961] Step 3:

[0962] The device uses voice recognition software to convert the received voice data into text data, which results in the text data "Research the world economy in 2023 and create a document."

[0963] Step 4:

[0964] The terminal transmits the converted text data to the server using an appropriate communication protocol (e.g., HTTPS).

[0965] Step 5:

[0966] The server receives the text data and simultaneously analyzes the emotions contained in the user's voice input using an emotion engine, recognizing, for example, impatience, excitement, calmness, etc.

[0967] Step 6:

[0968] The server uses generative AI to collect necessary information based on the received text data and the recognized emotions, such as the latest information on the global economy in 2023, from external databases and the internet.

[0969] Step 7:

[0970] The server organizes the collected information and creates reports and materials that reflect the emotional state recognized by the emotion engine. For example, if the user is excited, the server creates materials that prioritize providing detailed and prompt information.

[0971] Step 8:

[0972] The server uses text-to-speech (TTS) technology to generate emotionally rich voice data based on the created materials, and the voice is played back with intonation and speed that corresponds to the user's emotional state.

[0973] Step 9:

[0974] The server then transmits the generated voice data to the terminal, again using a communication protocol.

[0975] Step 10:

[0976] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[0977] This series of processing steps allows users to efficiently and effectively request information research and document creation, and receive the results in an emotionally rich voice. This system significantly reduces time and effort, and ensures high-quality information transmission.

[0978] Example 2

[0979] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0980] Conventional voice input systems provide information and create materials without considering the user's emotions, often generating results that do not match the user's intentions or emotions. Furthermore, information retrieval and material creation may not be performed quickly, resulting in a decrease in user satisfaction. Furthermore, the generated materials cannot be converted into emotionally rich speech, which is a problem that prevents the presentation from being fully effective for the user.

[0981] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for recognizing the user's emotion based on the received text data, means for collecting information based on the recognized emotion information and the text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, and means for playing back the voice. This makes it possible to quickly and effectively collect information and create materials while taking the user's emotions into consideration, and to improve the effectiveness of presentations by providing information to the user in emotive voice.

[0982] "Means for receiving voice input from a user" refers to a device or mechanism that receives voice when a user gives instructions or asks a question by voice.

[0983] "Means for converting said voice input into text data" refers to software or algorithms for analyzing received voice input and converting it into a corresponding text format.

[0984] "Means for transmitting converted text data to a server" refers to a device or software for transmitting the text data converted from speech to a server using an appropriate communication protocol.

[0985] "Means for recognizing a user's emotions based on received text data" refers to algorithms or software for analyzing and recognizing a user's emotions or mental state from text data.

[0986] "Means for collecting information based on recognized emotional information and text data" refers to software or algorithms that use the recognized emotional state of a user and text data to collect the required information from external databases or other sources.

[0987] "Means for generating materials from collected information" refers to software or algorithms for organizing collected information into an appropriate format and structuring it into materials.

[0988] "Means for converting generated materials into emotive speech" refers to software or algorithms for generating speech data that makes full use of emotion and intonation based on the content of the created materials.

[0989] The "means for reproducing the audio" refers to a device or mechanism that reproduces the generated audio data on a terminal and conveys it to the user as audio.

[0990] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system includes a function that recognizes the user's emotions and adjusts the response content and tone of the voice output. Specific embodiments of the system are described below.

[0991] System configuration

[0992] This system is realized through the cooperation of a user terminal, a server, and an emotion recognition engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[0993] Hardware and Software Configuration

[0994] User Device: A device that has a microphone for receiving voice input, a processor for speech recognition, and a speaker for audio playback, using speech recognition software (e.g., Google Cloud Speech-to-Text).

[0995] Server: A device equipped with a powerful processor for information gathering, document creation, emotion recognition, and speech conversion. It uses GPT-4 or similar as a generative AI model, and general-purpose emotion analysis software (e.g., IBM Watson) as an emotion recognition engine. Google Cloud Text-to-Speech is used for speech synthesis.

[0996] Operational Overview

[0997] The operation of this system is as follows.

[0998] Voice input and text conversion: When a user speaks commands or asks questions, the user device captures the speech and converts it into text data using voice recognition software.

[0999] Transmission of text data: The converted text data is transmitted from the user terminal to the server using a secure communication protocol (e.g., HTTPS).

[1000] Emotion recognition: The server inputs the received text data into an emotion recognition engine to recognize the user's emotions, for example, determining whether the user is anxious or excited.

[1001] Information collection and document creation: The server uses a generative AI model to collect information and create documents based on the recognized emotion information and text data.

[1002] Emotionally rich speech generation: The created materials are converted into speech data that reflects the user's emotions. The speech synthesis software adjusts intonation and speed according to the emotion.

[1003] Transmission and playback of voice data: The generated voice data is transmitted from the server to the user terminal and played back on the user terminal.

[1004] Specific examples

[1005] Example 1: Requesting an economic report

[1006] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[1007] 2. The user device converts this voice instruction into the text "Research the world economy in 2023 and create materials."

[1008] 3. The user terminal sends the text data to the server.

[1009] 4. The server uses an emotion recognition engine to recognize emotions such as impatience and excitement from the user's voice.

[1010] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[1011] 6. The server generates the following statement in an emotive voice, with a calm and smooth intonation: "The global economy in 2023 will be extremely unstable..."

[1012] 7. The server sends the audio data to the terminal, and the user terminal plays it and reports it to the user.

[1013] This allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, saving time and effort while enabling high-quality information transmission.

[1014] Example prompt statement:

[1015] Generate the output speech when the user requests, "Find the latest statistics on climate change and create a report," but in a calm tone.

[1016] By using such prompts, the generative AI model can create and voice materials that meet the user's needs.

[1017] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1018] Step 1:

[1019] The user speaks instructions and questions into the terminal. This voice input is the starting point of the system, and specifically, the input is something like "Research the world economy in 2023 and create materials." This voice data becomes the raw data that will be processed in the next step.

[1020] Step 2:

[1021] When the device receives a user's voice input, it uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. The input here is voice data, and the output is the corresponding text data. The voice waveform data is analyzed, and a language model is applied based on the phonetic patterns to convert it into a string of characters.

[1022] Step 3:

[1023] The terminal sends the converted text data to the server using a secure communication protocol such as HTTPS, and the converted text data is sent as input to the server. During this transmission process, the data is encrypted and divided into packets.

[1024] Step 4:

[1025] The server analyzes the received text data and inputs it into an emotion recognition engine. The emotion recognition engine (e.g., IBM Watson) extracts the user's emotion from the input text. At this stage, the input is text data, and the output is emotion data such as "anxiety" or "excitement." Natural language processing technology is used to infer emotions from context and vocabulary.

[1026] Step 5:

[1027] The server uses a generative AI model (e.g., OpenAI's GPT-4) to collect information and create materials based on the recognized emotional information and text data. The input here is emotional information and text data, and the output is specific materials (in text format). Information is collected using an external database API, and the generative AI model summarizes and analyzes the collected data.

[1028] Step 6:

[1029] The server converts the created material into emotionally rich audio data using text-to-speech (TTS) technology (e.g., Google Cloud Text-to-Speech). The input is the text data of the material, and the output is audio data. Processing is performed to adjust intonation and speed based on the emotional information.

[1030] Step 7:

[1031] The server sends the generated voice data to the user's device. The device plays the received voice data and conveys the content to the user. The input here is voice data, and the output is voice playback to the user. An emotionally rich voice is played through the device's speaker.

[1032] Through these processing steps, users can efficiently gather information and receive emotionally appropriate presentations, thereby saving time and effort and enabling high-quality information transmission.

[1033] (Application example 2)

[1034] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1035] In modern factories, workers need to report problems in real time and receive fast, accurate solutions. However, with current systems, it can take a long time from the time a problem is reported to the time a solution is provided, which can lead to work stoppages and delays. Additionally, solution presentations often lack emotional explanations, making it difficult for workers to fully understand the content and take prompt action.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1037] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, means for adjusting the generated voice to match the user's emotional state, and means for playing back the voice, thereby enabling workers to report problems in real time, provide problem-solving methods by referring to a past database, and explain the solutions in emotive voice.

[1038] A "means for receiving voice input from a user" is a device that captures commands or questions spoken by a user through a voice input device such as a microphone.

[1039] The "means for converting the voice input into text data" refers to software or a system that uses voice recognition technology to convert the user's voice input into text information.

[1040] The "means for transmitting the converted text data to the server" refers to a protocol and device for transferring the text data to the server using a communication network such as the Internet.

[1041] The "means for collecting information based on received text data" is a system that collects related information from internal and external databases and information sources according to the content of the text data.

[1042] "Means for generating collected information as materials" refers to a system that organizes collected information and creates reports and documents in a format that is easy for users to understand.

[1043] The "means for converting generated materials into emotive speech" is a system that converts the created text material into speech using text-to-speech technology and adds intonation and speed that match the user's emotions.

[1044] "Means for adjusting the generated voice to match the emotional state of the user" refers to a technology that analyzes the user's emotions and adjusts the tone, speed, and intonation of the voice as needed.

[1045] The "means for reproducing the audio" is a device that allows the user to listen to the generated audio data through an audio output device such as a speaker.

[1046] The system of the present invention aims to manage production processes and support problem solving in factories. The system includes a user terminal, a server, an emotion engine, and a voice output device.

[1047] System Configuration

[1048] 1. User Device

[1049] This includes microphones installed on smartphones and factory robots as voice input devices.

[1050] As a speech recognition software, it uses Google Cloud Speech-to-Text API to convert user voice input into text data.

[1051] 2. Server

[1052] The text data is received and the user's emotional state is analyzed using IBM Watson Tone Analyzer.

[1053] OpenAI's GPT-4 is used as the generative AI model to collect relevant information and generate materials based on the received text data.

[1054] Based on the collected information, the material is converted into emotive speech using Amazon Polly.

[1055] The audio data is sent to the terminal and a playback instruction is issued.

[1056] 3. Emotion Engine

[1057] The server uses IBM Watson Tone Analyzer to recognize the user's emotions contained in the voice input and adjust the tone of the generated voice.

[1058] 4. Audio Output Device

[1059] These include speakers built into smartphones and factory robots, which play back audio data to users and provide them with work instructions and problem-solving methods.

[1060] Specific examples

[1061] If a factory worker notices a problem on the production line, they can use the system to report the problem in real time. For example, the worker can voice-record the following:

[1062] "An error has occurred on production line A. Please investigate the cause."

[1063] This voice input is captured by the microphone of a smartphone or factory robot and converted into text data using the Google Cloud Speech-to-Text API. This text data is then sent to a server via the internet, where the user's emotional state is analyzed using IBM Watson Tone Analyzer. OpenAI's GPT-4 is then used to collect relevant information from a database and generate a document proposing solutions to the problem.

[1064] The generated materials are converted into speech using Amazon Polly, with the tone adjusted to match the worker's emotion. The speech data is then sent to a terminal and played back through the worker's smartphone or the factory robot's speaker.

[1065] Prompt Sentence Examples

[1066] An error has occurred on production line A. Please prepare a report in a matter-of-fact manner, including any similar errors that have occurred in the past and solutions to those errors.

[1067] In this manner, the system of the present invention provides a means for rapid and effective problem resolution within the factory.

[1068] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1069] Step 1:

[1070] Acquiring voice input

[1071] The user issues instructions or asks questions by voice into the microphone of a smartphone or factory robot. For example, they might type, "An error has occurred on production line A. Please investigate the cause."

[1072] Input: User's voice commands

[1073] Output: Audio data collected by the microphone

[1074] Step 2:

[1075] Voice to text conversion

[1076] The device receives the collected voice data and converts it into text data using the Google Cloud Speech-to-Text API.

[1077] Input: Audio data

[1078] Output: Converted text data

[1079] Step 3:

[1080] Sending text data to the server

[1081] The terminal sends the converted text data to the server using the HTTPS protocol.

[1082] Input: Text data

[1083] Output: Text data sent to the server

[1084] Step 4:

[1085] emotion recognition

[1086] The server inputs the received text data into the IBM Watson Tone Analyzer to analyze the user's emotional state.

[1087] Input: Text data

[1088] Output: Parsed emotion information

[1089] Step 5:

[1090] Information gathering and document creation

[1091] The server uses OpenAI's GPT-4 to collect relevant information from a database using the analyzed emotional information and text data, and generates a document containing specific solutions, using the prompt, "An error has occurred on production line A. Please prepare a report in a calm tone, including past similar errors and solutions."

[1092] Input: Analyzed emotion information, text data

[1093] Output: Generated material

[1094] Step 6:

[1095] Emotional voice conversion

[1096] The server converts the generated material into audio data using Amazon Polly, adjusting the intonation and speed to match the user's emotional state.

[1097] Input: Generated material

[1098] Output: Emotionally rich audio data

[1099] Step 7:

[1100] Sending audio data to the device

[1101] The server sends the generated audio data to the terminal using the HTTPS protocol.

[1102] Input: Emotionally rich speech data

[1103] Output: Audio data sent to the device

[1104] Step 8:

[1105] Playing audio data

[1106] The terminal reproduces the received audio data through a speaker to convey information to the user.

[1107] Input: Transmitted audio data

[1108] Output: The audio data played to the user

[1109] This process allows users to report problems in the factory in real time and receive fast, accurate solutions.

[1110] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1111] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1112] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1113] [Fourth embodiment]

[1114] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1115] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1116] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1117] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1118] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1119] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1120] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1121] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1122] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1123] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1124] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1125] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1126] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1127] The present invention is a system that allows a user to request information research or the creation of materials through voice input, and then presents the created materials in an expressive voice. Specific embodiments of the present invention will be described below.

[1128] System Configuration

[1129] This system is realized by the cooperation of the user terminal and the server. The user terminal mainly handles voice input and processing, while the server is responsible for collecting information and creating materials.

[1130] Program operation overview

[1131] 1. Acquiring voice input

[1132] The user issues voice instructions to the device, such as "Research the world economy in 2023 and create a document."

[1133] 2. Voice to text conversion

[1134] The device receives voice input and converts it to text using speech recognition software, which processes the voice data as text.

[1135] 3. Sending text data to the server

[1136] The device then sends the converted text data to the server, using an appropriate API or similar.

[1137] 4. Information gathering and document preparation

[1138] The server uses generative AI based on the received text data to collect information, for example, by retrieving necessary information from external databases or the internet and creating documents based on that information.

[1139] 5. Conversion to emotionally rich speech

[1140] The server converts the generated material into emotive speech using text-to-speech (TTS) technology, which translates the material into natural intonation.

[1141] 6. Sending audio data to the device and playing it back

[1142] After generating the audio data, the server transmits it to the terminal, which then plays the received audio data in a format that presents it to the user.

[1143] Specific examples

[1144] Example 1: Requesting an economic report

[1145] 1. The user says, "Research the world economy in 2023 and create a document."

[1146] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[1147] 3. The device sends this text data to the server.

[1148] 4. The server uses generative AI to collect the latest information on the global economy in 2023 and create a report.

[1149] 5. The server generates an emotive voice based on this report, for example, "The global economy in 2023 will be very unstable..."

[1150] 6. The server sends the audio file to the terminal, which plays it and reports it to the user.

[1151] In this way, by utilizing this system, users can easily request information research and document preparation, and receive the results in high-quality audio, enabling efficient and effective information gathering, document preparation, and presentations.

[1152] The processing flow will be explained below.

[1153] Step 1:

[1154] The user speaks to the device, for example, "Research the world economy in 2023 and create a document."

[1155] Step 2:

[1156] The device receives the user's voice input through a microphone and then uses speech recognition software to convert the received speech into text data.

[1157] Step 3:

[1158] The device sends the converted text data to the server, using an appropriate API to send the text data to the server over the network.

[1159] Step 4:

[1160] The server receives the text data and passes it to the generation AI, which then collects the necessary information based on the text data and creates materials. For example, it may collect information about the world economy in 2023 from external databases or the internet.

[1161] Step 5:

[1162] The server creates documents based on the collected information, which are organized in text format and clearly describe the content to be provided to users.

[1163] Step 6:

[1164] The server converts the created materials into emotionally rich speech, using text-to-speech (TTS) technology to generate speech data incorporating natural intonation and emotional expressions.

[1165] Step 7:

[1166] The server then sends the generated voice data to the device, which is also sent via the network via the API.

[1167] Step 8:

[1168] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[1169] This series of processing steps allows users to efficiently request information research and document creation, and receive the results in an expressive voice. This system significantly reduces users' time and effort, and enables high-quality presentations.

[1170] Example 1

[1171] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1172] In recent years, there has been a demand for more efficient information research and document creation, but the time and effort required for these tasks remains a challenge. In particular, there is a demand for systems that can efficiently and automatically perform a series of tasks, including collecting information using voice input, generating documents, and then presenting those documents with expressive voice. However, conventional systems require some of these processes to be performed manually, which can result in reduced efficiency.

[1173] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1174] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text information, and means for transmitting the converted text information to a data processing device, thereby enabling efficient information gathering and document generation based on the voice input, and the results to be presented in an expressive voice.

[1175] "User" refers to a person who requests information research or document creation through voice input.

[1176] "Voice input" refers to the voice that a user speaks into a terminal, and is a means for making requests for information research or document creation by voice.

[1177] "Text information" refers to text data converted from voice input using voice recognition technology.

[1178] "Data processing device" refers to a computer system that receives text information and performs processes such as information collection, document creation, and audio data generation.

[1179] "Data" refers to information such as documents, numbers, statistics, and images obtained as a result of information gathering, and serves as the basis for creating materials.

[1180] "Document" refers to a material generated based on information collected by a data processing device and expressed in text format.

[1181] "Audio data" refers to data in which a document has been converted into expressive audio, and refers to an audio file that is output by a playback device.

[1182] A "playback device" refers to equipment or software that allows users to listen to audio data, and includes, for example, speakers and media players.

[1183] This invention is a system in which a user can request information research or document creation through voice input, and the created document is then presented in an expressive voice. This system is realized mainly through the cooperation of a user terminal and a server.

[1184] System Configuration

[1185] This system consists of the following components:

[1186] User device: A device that receives voice input and converts it into text, using good voice recognition software (e.g., Google Speech-to-Text API).

[1187] Server: A device that collects information, generates documents, and generates emotive voice data based on text information. Here, it uses generative AI models (e.g., OpenAI GPT-4) and speech synthesis technology (e.g., Amazon Polly).

[1188] Communication method: API communication (e.g., HTTP POST) is used to send and receive data between the user terminal and the server.

[1189] Processing flow

[1190] 1. Acquisition of voice input: The user speaks into the device to request specific research or document creation. The device uses the built-in microphone to capture the voice.

[1191] 2. Speech-to-text conversion: The device uses the Google Speech-to-Text API to convert speech into text, turning the speech data into processable text.

[1192] 3. Sending text data to the server: The device sends the converted text data to the server via an HTTP POST request, which includes the destination URL and authentication information.

[1193] 4. Information collection and document creation: The server analyzes the received text data and uses a generative AI model (GPT-4) to collect the necessary information and create documents. The necessary information is collected from external databases and the Internet.

[1194] 5. Conversion to emotive speech: The server sends the generated material to Amazon Polly, which converts it into natural, emotive speech data.

[1195] 6. Sending and playing back audio data to the terminal: The server sends the generated audio file to the terminal, and the terminal plays back the audio data in a way that presents it to the user.

[1196] Specific examples

[1197] Example 1: Requesting an economic report

[1198] 1. The user says, "Research the world economy in 2023 and create a document."

[1199] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[1200] 3. The device sends this text data to the server.

[1201] 4. The server uses a generative AI model (GPT-4) to collect the latest information on the global economy in 2023 and create a report.

[1202] 5. The server generates emotive audio based on this report, creating a presentation audio with the following content, for example, "The global economy in 2023 will be extremely unstable..."

[1203] 6. The server sends the audio data to the terminal, which plays it and reports it to the user.

[1204] Prompt Sentence Examples

[1205] "Research and write a detailed report on the global economy in 2023. Then, turn that report into an audio presentation."

[1206] "Gather information about the latest impacts of climate change, create materials and give an emotive presentation."

[1207] By utilizing this system, users can efficiently conduct information research, create documents, and make presentations.

[1208] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1209] Step 1:

[1210] The user speaks to the device to request specific research or document creation. The input is the user's voice, such as "Research the world economy in 2023 and create a document." The device picks up this voice using its built-in microphone.

[1211] Step 2:

[1212] The device sends the acquired voice data to the Google Speech-to-Text API and converts the voice into text information. The input is the voice data acquired in step 1, and text information is obtained by sending this voice data to the API. The output is text information in text format.

[1213] Specific behavior:

[1214] 1. Obtain the audio file recorded by the device.

[1215] 2. Send the acquired audio file to the Google Speech-to-Text API.

[1216] 3. Save the text information returned as an API response.

[1217] Step 3:

[1218] The terminal sends the converted text information to the server via an HTTP POST request. The input is the text information obtained in step 2, and an HTTP request is created to send this information to the server. The output is a response indicating that the request was successful.

[1219] Specific behavior:

[1220] 1. Convert the text information into JSON format and set it as the payload of the HTTP POST request.

[1221] 2. Set authentication information and destination URL in the request header.

[1222] 3. Send an HTTP request to the server and receive a response.

[1223] Step 4:

[1224] The server analyzes the received text information, collects the necessary information using a generative AI model (e.g., GPT-4), and generates the document. The input is the text information sent in step 3, and based on this information, it sends a prompt to the AI ​​model. The output is the generated document.

[1225] Specific behavior:

[1226] 1. The server analyzes the received text information.

[1227] 2. Generate a prompt for GPT-4 to begin collecting information.

[1228] 3. Create materials (documents) based on the returned information.

[1229] Step 5:

[1230] The server sends the created material to Amazon Polly, which converts it into natural, emotive speech data. The input is the material generated in step 4, which is sent to a text-to-speech API to be converted into speech. The output is an audio file.

[1231] Specific behavior:

[1232] 1. Save the generated material in text format.

[1233] 2. Send this text data to the Amazon Polly API.

[1234] 3. Save the audio file (e.g., MP3 format) returned as an API response.

[1235] Step 6:

[1236] The server sends the generated audio file to the terminal. The terminal receives this audio data and plays it in a form that is presented to the user. The input is the audio file generated in step 5, which is sent from the server to the terminal. The output is the playback of the audio file.

[1237] Specific behavior:

[1238] 1. The server sends the audio file in an HTTP response.

[1239] 2. The device saves the received audio file locally.

[1240] 3. Play the saved audio file in a media player and let the user listen to it.

[1241] Through this series of processes, the user can automatically conduct research based on the content specified by voice, create materials, and receive the content in an expressive voice.

[1242] (Application example 1)

[1243] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1244] In conventional self-driving vehicles, when a user requests information by voice, it is difficult to search for information or generate materials in real time. It is also difficult to present the acquired information in an expressive voice. As a result, users often experience inconvenience because they are unable to immediately obtain useful information about their destination or current situation. The present invention aims to solve these problems and provide a system that enables real-time, expressive voice presentations in self-driving vehicles.

[1245] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1246] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into expressive voice, means for use in an autonomous vehicle, means for providing information and generating materials in real time, means for generating materials using a generative AI model, and means for transmitting and playing voice to a terminal in the autonomous vehicle. This enables a user to obtain necessary information in real time in an autonomous vehicle and receive the information in expressive voice.

[1247] "Means for receiving voice input" refers to a device or software that recognizes the voice uttered by the user and inputs the voice data into the system.

[1248] The "means for converting speech input into text data" refers to a device or software that analyzes received speech data and outputs the resulting text data.

[1249] The "means for transmitting the converted text data to the server" refers to a communication device or software for sending the text data to the server via a network.

[1250] "Information gathering means" refers to a device or software that searches and retrieves the required information from a server or other database.

[1251] "Means for generating materials" refers to devices or software that process collected information and compile it into a format that is easy for users to use.

[1252] The "means for converting into emotionally rich speech" is a device or software that generates speech with natural intonation including emotional expressions based on the generated text data.

[1253] The "means for reproducing audio" is a device such as a speaker for reproducing the generated audio data as actual sound.

[1254] "In-vehicle means" means any device or software that is installed and operates within an automated vehicle.

[1255] "Means for providing information and generating materials in real time" refers to a device or software for instantly collecting information and generating materials in response to user instructions.

[1256] A "means for generating materials using a generative AI model" is a device or software that uses advanced artificial intelligence to collect information and generate materials.

[1257] The "means for transmitting audio to a terminal and playing it back" refers to a device or software that transmits audio data generated by a server to a terminal and plays back the audio on the terminal.

[1258] The present invention is a system that provides real-time information within an autonomous vehicle, generates materials, and reports them to the user in an expressive voice. The system receives voice input, converts the content into text data, and sends it to a server. The server collects information based on the received text data and generates materials using a generative AI model. The generated materials are converted into expressive voice, and the voice data is sent to a terminal. Finally, the terminal plays the received voice data for the user.

[1259] About program processing

[1260] 1. Acquiring voice input:

[1261] The device inside the car captures the user's voice. This uses voice recognition software (e.g., Google Cloud Speech-to-Text) to capture the user's voice as digital data.

[1262] 2. Voice to text conversion:

[1263] The captured voice data is analyzed and converted into corresponding text data. At this stage, speech recognition software converts speech into text.

[1264] 3. Sending text data:

[1265] The converted text data is sent to the server using the vehicle's communication module, where a RESTful API is used as the communication method.

[1266] 4. Information gathering and data generation:

[1267] The server uses a generative AI model (such as OpenAI's GPT-4) to collect information based on the received text data. Specifically, it searches external databases and information on the internet to collect and analyze the necessary data.

[1268] Based on the collected information, materials are generated. A generative AI model is used to generate these materials.

[1269] 5. Emotional voice conversion:

[1270] The material is converted into emotive audio using text-to-speech (TTS) technology, such as the TTS capabilities of Azure Cognitive Services.

[1271] 6. Playing audio data:

[1272] The server transmits the generated voice data to an in-vehicle terminal, which then plays the voice data through a speaker.

[1273] Specific examples

[1274] For example, suppose a user speaks to an in-car device, saying, "Tell me about recommended restaurants near the next rest stop." This voice input is converted to text using Google Cloud Speech-to-Text and sent to the server. The server uses OpenAI's GPT-4 to collect information about recommended restaurants near the next rest stop from the internet. The collected information is compiled into a document, and the content is converted into emotive speech using Azure Cognitive Services' TTS technology. Finally, the speech is sent to the in-car device and played over the speaker. An example of a prompt sentence is, "Tell me about recommended restaurants near the next rest stop." This system allows users to obtain the information they need in real time and receive it in emotive speech.

[1275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1276] Step 1:

[1277] The user inputs voice into the terminal inside the autonomous vehicle. Specifically, they speak through the microphone, saying, "Tell me about a recommended restaurant near the next rest stop." This voice data is captured by the terminal. The input is the user's voice, and the output is analog voice data.

[1278] Step 2:

[1279] The device receives voice input and converts the voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). Specifically, the voice recognition engine analyzes the sound waveform and generates a corresponding string of characters. The input is voice data, and the output is text data.

[1280] Step 3:

[1281] The device sends the content converted into text data to the server. A RESTful API is used as the communication method, and the text data is sent as an HTTP POST request. Specifically, the device sends the generated text data to the server in JSON format. The input is text data, and the output is data sent to the server.

[1282] Step 4:

[1283] The server uses a generative AI model (such as OpenAI's GPT-4) to collect the necessary information based on the received text data. Specifically, it searches for and retrieves relevant information from the internet and internal databases. The input is text data, and the output is the collected information.

[1284] Step 5:

[1285] The server generates documents based on the collected information. Here too, a generative AI model is used to organize the information and summarize it in a form that is easy for humans to understand. Specific operations include generating sentences, creating graphs, and formatting lists. The input is the collected information, and the output is the generated documents.

[1286] Step 6:

[1287] The server uses text-to-speech (TTS) technology (e.g., Azure Cognitive Services) to convert the generated material into emotive speech. Specifically, the TTS engine takes text as input and generates speech data containing emotional expressions. The input is the generated text material, and the output is speech data.

[1288] Step 7:

[1289] The server sends the generated audio data to the device, which receives it and plays it through the speaker. Specifically, the device downloads the audio file and plays it through the audio output device. The input is audio data, and the output is audio data playback for the user.

[1290] Example: Prompt sentence

[1291] For example, a user may input a prompt such as "Tell me about a recommended restaurant near the next rest stop." The entire system operates based on this prompt.

[1292] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1293] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system also includes a function for adjusting the content of responses and the tone of the voice output by combining it with an emotion engine that recognizes the user's emotions. Specific embodiments of the system are described below.

[1294] System Configuration

[1295] This system is realized through the cooperation of a user terminal, a server, and an emotion engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[1296] Program operation overview

[1297] 1. Acquiring voice input

[1298] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[1299] 2. Voice to text conversion

[1300] The device receives the user's voice input and converts it into text data using speech recognition software.

[1301] 3. Sending text data to the server

[1302] The terminal sends the converted text data to the server, which then sends the data using the appropriate communication protocol.

[1303] 4. Emotion recognition

[1304] The server utilizes an emotion engine to recognize emotions contained in the user's voice input, which allows for real-time analysis of the user's emotional state when asking a question.

[1305] 5. Information gathering and document preparation

[1306] The server uses generative AI to gather information based on the received text data and the user's perceived emotions. For example, it retrieves the latest information on the global economy in 2023 from an external database. The user's emotional state is reflected in the tone and content of the information.

[1307] 6. Emotional voice conversion

[1308] The server uses text-to-speech (TTS) technology to convert the created material into expressive speech, with intonation and speed that reflects the user's emotions.

[1309] 7. Sending audio data to the device and playing it back

[1310] The server transmits the generated voice data to the terminal, which then plays back the received voice data and conveys the content of the material to the user in an emotional way.

[1311] Specific examples

[1312] Example 1: Requesting an economic report

[1313] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[1314] 2. The device converts this voice instruction into text: "Research the world economy in 2023 and create a document."

[1315] 3. The device sends the text data to the server.

[1316] 4. The server uses an emotion engine to recognize emotions such as impatience and excitement from the user's voice.

[1317] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[1318] 6. The server generates the following statement in an emotive voice: "The global economy in 2023 will be extremely unstable..." but delivers it smoothly and calmly.

[1319] 7. The server sends the audio data to the terminal, which plays it and reports it to the user.

[1320] This system allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, thereby saving time and effort and enabling the transmission of high-quality information.

[1321] The processing flow will be explained below.

[1322] Step 1:

[1323] The user speaks commands and questions into the device, for example, "Research the world economy in 2023 and create a document."

[1324] Step 2:

[1325] The device receives the user's voice input through a microphone and passes the voice data to voice recognition software.

[1326] Step 3:

[1327] The device uses voice recognition software to convert the received voice data into text data, which results in the text data "Research the world economy in 2023 and create a document."

[1328] Step 4:

[1329] The terminal transmits the converted text data to the server using an appropriate communication protocol (e.g., HTTPS).

[1330] Step 5:

[1331] The server receives the text data and simultaneously analyzes the emotions contained in the user's voice input using an emotion engine, recognizing, for example, impatience, excitement, calmness, etc.

[1332] Step 6:

[1333] The server uses generative AI to collect necessary information based on the received text data and the recognized emotions, such as the latest information on the global economy in 2023, from external databases and the internet.

[1334] Step 7:

[1335] The server organizes the collected information and creates reports and materials that reflect the emotional state recognized by the emotion engine. For example, if the user is excited, the server creates materials that prioritize providing detailed and prompt information.

[1336] Step 8:

[1337] The server uses text-to-speech (TTS) technology to generate emotionally rich voice data based on the created materials, and the voice is played back with intonation and speed that corresponds to the user's emotional state.

[1338] Step 9:

[1339] The server then transmits the generated voice data to the terminal, again using a communication protocol.

[1340] Step 10:

[1341] The terminal plays back the received audio data, and the user can receive information or a presentation of materials based on the requested content through the played back audio.

[1342] This series of processing steps allows users to efficiently and effectively request information research and document creation, and receive the results in an emotionally rich voice. This system significantly reduces time and effort, and ensures high-quality information transmission.

[1343] Example 2

[1344] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1345] Conventional voice input systems provide information and create materials without considering the user's emotions, often generating results that do not match the user's intentions or emotions. Furthermore, information retrieval and material creation may not be performed quickly, resulting in a decrease in user satisfaction. Furthermore, the generated materials cannot be converted into emotionally rich speech, which is a problem that prevents the presentation from being fully effective for the user.

[1346] The specification process by the specification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving a user's voice input, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for recognizing the user's emotion based on the received text data, means for collecting information based on the recognized emotion information and the text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, and means for playing back the voice. This makes it possible to quickly and effectively collect information and create materials while taking the user's emotions into consideration, and to improve the effectiveness of presentations by providing information to the user in emotive voice.

[1347] "Means for receiving voice input from a user" refers to a device or mechanism that receives voice when a user gives instructions or asks a question by voice.

[1348] "Means for converting said voice input into text data" refers to software or algorithms for analyzing received voice input and converting it into a corresponding text format.

[1349] "Means for transmitting converted text data to a server" refers to a device or software for transmitting the text data converted from speech to a server using an appropriate communication protocol.

[1350] "Means for recognizing a user's emotions based on received text data" refers to algorithms or software for analyzing and recognizing a user's emotions or mental state from text data.

[1351] "Means for collecting information based on recognized emotional information and text data" refers to software or algorithms that use the recognized emotional state of a user and text data to collect the required information from external databases or other sources.

[1352] "Means for generating materials from collected information" refers to software or algorithms for organizing collected information into an appropriate format and structuring it into materials.

[1353] "Means for converting generated materials into emotive speech" refers to software or algorithms for generating speech data that makes full use of emotion and intonation based on the content of the created materials.

[1354] The "means for reproducing the audio" refers to a device or mechanism that reproduces the generated audio data on a terminal and conveys it to the user as audio.

[1355] The present invention relates to a system that allows a user to request information research or document creation by voice input, and then presents the created document in an emotionally rich voice. Furthermore, this system includes a function that recognizes the user's emotions and adjusts the response content and tone of the voice output. Specific embodiments of the system are described below.

[1356] System configuration

[1357] This system is realized through the cooperation of a user terminal, a server, and an emotion recognition engine. The user terminal mainly handles voice input and processing, while the server is responsible for information collection, document creation, and emotion recognition.

[1358] Hardware and Software Configuration

[1359] User Device: A device that has a microphone for receiving voice input, a processor for speech recognition, and a speaker for audio playback, using speech recognition software (e.g., Google Cloud Speech-to-Text).

[1360] Server: A device equipped with a powerful processor for information gathering, document creation, emotion recognition, and speech conversion. It uses GPT-4 or similar as a generative AI model, and general-purpose emotion analysis software (e.g., IBM Watson) as an emotion recognition engine. Google Cloud Text-to-Speech is used for speech synthesis.

[1361] Operational Overview

[1362] The operation of this system is as follows.

[1363] Voice input and text conversion: When a user speaks commands or asks questions, the user device captures the speech and converts it into text data using voice recognition software.

[1364] Transmission of text data: The converted text data is transmitted from the user terminal to the server using a secure communication protocol (e.g., HTTPS).

[1365] Emotion recognition: The server inputs the received text data into an emotion recognition engine to recognize the user's emotions, for example, determining whether the user is anxious or excited.

[1366] Information collection and document creation: The server uses a generative AI model to collect information and create documents based on the recognized emotion information and text data.

[1367] Emotionally rich speech generation: The created materials are converted into speech data that reflects the user's emotions. The speech synthesis software adjusts intonation and speed according to the emotion.

[1368] Transmission and playback of voice data: The generated voice data is transmitted from the server to the user terminal and played back on the user terminal.

[1369] Specific examples

[1370] Example 1: Requesting an economic report

[1371] 1. The user types in "Research the world economy in 2023 and create a document," expressing feelings of impatience and excitement.

[1372] 2. The user device converts this voice instruction into the text "Research the world economy in 2023 and create materials."

[1373] 3. The user terminal sends the text data to the server.

[1374] 4. The server uses an emotion recognition engine to recognize emotions such as impatience and excitement from the user's voice.

[1375] 5. The server creates materials that reflect that emotional state and emphasize providing information quickly and in detail.

[1376] 6. The server generates the following statement in an emotive voice, with a calm and smooth intonation: "The global economy in 2023 will be extremely unstable..."

[1377] 7. The server sends the audio data to the terminal, and the user terminal plays it and reports it to the user.

[1378] This allows users to efficiently request information research and document creation, and receive the results in a format that suits their needs, saving time and effort while enabling high-quality information transmission.

[1379] Example prompt statement:

[1380] Generate the output speech when the user requests, "Find the latest statistics on climate change and create a report," but in a calm tone.

[1381] By using such prompts, the generative AI model can create and voice materials that meet the user's needs.

[1382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1383] Step 1:

[1384] The user speaks instructions and questions into the terminal. This voice input is the starting point of the system, and specifically, the input is something like "Research the world economy in 2023 and create materials." This voice data becomes the raw data that will be processed in the next step.

[1385] Step 2:

[1386] When the device receives a user's voice input, it uses speech recognition software (e.g., Google Cloud Speech-to-Text) to convert the voice into text data. The input here is voice data, and the output is the corresponding text data. The voice waveform data is analyzed, and a language model is applied based on the phonetic patterns to convert it into a string of characters.

[1387] Step 3:

[1388] The terminal sends the converted text data to the server using a secure communication protocol such as HTTPS, and the converted text data is sent as input to the server. During this transmission process, the data is encrypted and divided into packets.

[1389] Step 4:

[1390] The server analyzes the received text data and inputs it into an emotion recognition engine. The emotion recognition engine (e.g., IBM Watson) extracts the user's emotion from the input text. At this stage, the input is text data, and the output is emotion data such as "anxiety" or "excitement." Natural language processing technology is used to infer emotions from context and vocabulary.

[1391] Step 5:

[1392] The server uses a generative AI model (e.g., OpenAI's GPT-4) to collect information and create materials based on the recognized emotional information and text data. The input here is emotional information and text data, and the output is specific materials (in text format). Information is collected using an external database API, and the generative AI model summarizes and analyzes the collected data.

[1393] Step 6:

[1394] The server converts the created material into emotionally rich audio data using text-to-speech (TTS) technology (e.g., Google Cloud Text-to-Speech). The input is the text data of the material, and the output is audio data. Processing is performed to adjust intonation and speed based on the emotional information.

[1395] Step 7:

[1396] The server sends the generated voice data to the user's device. The device plays the received voice data and conveys the content to the user. The input here is voice data, and the output is voice playback to the user. An emotionally rich voice is played through the device's speaker.

[1397] Through these processing steps, users can efficiently gather information and receive emotionally appropriate presentations, thereby saving time and effort and enabling high-quality information transmission.

[1398] (Application example 2)

[1399] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1400] In modern factories, workers need to report problems in real time and receive fast, accurate solutions. However, with current systems, it can take a long time from the time a problem is reported to the time a solution is provided, which can lead to work stoppages and delays. Additionally, solution presentations often lack emotional explanations, making it difficult for workers to fully understand the content and take prompt action.

[1401] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1402] In this invention, the server includes means for receiving voice input from a user, means for converting the voice input into text data, means for transmitting the converted text data to the server, means for collecting information based on the received text data, means for generating the collected information as materials, means for converting the generated materials into emotive voice, means for adjusting the generated voice to match the user's emotional state, and means for playing back the voice, thereby enabling workers to report problems in real time, provide problem-solving methods by referring to a past database, and explain the solutions in emotive voice.

[1403] A "means for receiving voice input from a user" is a device that captures commands or questions spoken by a user through a voice input device such as a microphone.

[1404] The "means for converting the voice input into text data" refers to software or a system that uses voice recognition technology to convert the user's voice input into text information.

[1405] The "means for transmitting the converted text data to the server" refers to a protocol and device for transferring the text data to the server using a communication network such as the Internet.

[1406] The "means for collecting information based on received text data" is a system that collects related information from internal and external databases and information sources according to the content of the text data.

[1407] "Means for generating collected information as materials" refers to a system that organizes collected information and creates reports and documents in a format that is easy for users to understand.

[1408] The "means for converting generated materials into emotive speech" is a system that converts the created text material into speech using text-to-speech technology and adds intonation and speed that match the user's emotions.

[1409] "Means for adjusting the generated voice to match the emotional state of the user" refers to a technology that analyzes the user's emotions and adjusts the tone, speed, and intonation of the voice as needed.

[1410] The "means for reproducing the audio" is a device that allows the user to listen to the generated audio data through an audio output device such as a speaker.

[1411] The system of the present invention aims to manage production processes and support problem solving in factories. The system includes a user terminal, a server, an emotion engine, and a voice output device.

[1412] System Configuration

[1413] 1. User Device

[1414] This includes microphones installed on smartphones and factory robots as voice input devices.

[1415] As a speech recognition software, it uses Google Cloud Speech-to-Text API to convert user voice input into text data.

[1416] 2. Server

[1417] The text data is received and the user's emotional state is analyzed using IBM Watson Tone Analyzer.

[1418] OpenAI's GPT-4 is used as the generative AI model to collect relevant information and generate materials based on the received text data.

[1419] Based on the collected information, the material is converted into emotive speech using Amazon Polly.

[1420] The audio data is sent to the terminal and a playback instruction is issued.

[1421] 3. Emotion Engine

[1422] The server uses IBM Watson Tone Analyzer to recognize the user's emotions contained in the voice input and adjust the tone of the generated voice.

[1423] 4. Audio Output Device

[1424] These include speakers built into smartphones and factory robots, which play back audio data to users and provide them with work instructions and problem-solving methods.

[1425] Specific examples

[1426] If a factory worker notices a problem on the production line, they can use the system to report the problem in real time. For example, the worker can voice-record the following:

[1427] "An error has occurred on production line A. Please investigate the cause."

[1428] This voice input is captured by the microphone of a smartphone or factory robot and converted into text data using the Google Cloud Speech-to-Text API. This text data is then sent to a server via the internet, where the user's emotional state is analyzed using IBM Watson Tone Analyzer. OpenAI's GPT-4 is then used to collect relevant information from a database and generate a document proposing solutions to the problem.

[1429] The generated materials are converted into speech using Amazon Polly, with the tone adjusted to match the worker's emotion. The speech data is then sent to a terminal and played back through the worker's smartphone or the factory robot's speaker.

[1430] Prompt Sentence Examples

[1431] An error has occurred on production line A. Please prepare a report in a matter-of-fact manner, including any similar errors that have occurred in the past and solutions to those errors.

[1432] In this manner, the system of the present invention provides a means for rapid and effective problem resolution within the factory.

[1433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1434] Step 1:

[1435] Acquiring voice input

[1436] The user issues instructions or asks questions by voice into the microphone of a smartphone or factory robot. For example, they might type, "An error has occurred on production line A. Please investigate the cause."

[1437] Input: User's voice commands

[1438] Output: Audio data collected by the microphone

[1439] Step 2:

[1440] Voice to text conversion

[1441] The device receives the collected voice data and converts it into text data using the Google Cloud Speech-to-Text API.

[1442] Input: Audio data

[1443] Output: Converted text data

[1444] Step 3:

[1445] Sending text data to the server

[1446] The terminal sends the converted text data to the server using the HTTPS protocol.

[1447] Input: Text data

[1448] Output: Text data sent to the server

[1449] Step 4:

[1450] emotion recognition

[1451] The server inputs the received text data into the IBM Watson Tone Analyzer to analyze the user's emotional state.

[1452] Input: Text data

[1453] Output: Parsed emotion information

[1454] Step 5:

[1455] Information gathering and document creation

[1456] The server uses OpenAI's GPT-4 to collect relevant information from a database using the analyzed emotional information and text data, and generates a document containing specific solutions, using the prompt, "An error has occurred on production line A. Please prepare a report in a calm tone, including past similar errors and solutions."

[1457] Input: Analyzed emotion information, text data

[1458] Output: Generated material

[1459] Step 6:

[1460] Emotional voice conversion

[1461] The server converts the generated material into audio data using Amazon Polly, adjusting the intonation and speed to match the user's emotional state.

[1462] Input: Generated material

[1463] Output: Emotionally rich audio data

[1464] Step 7:

[1465] Sending audio data to the device

[1466] The server sends the generated audio data to the terminal using the HTTPS protocol.

[1467] Input: Emotionally rich speech data

[1468] Output: Audio data sent to the device

[1469] Step 8:

[1470] Playing audio data

[1471] The terminal reproduces the received audio data through a speaker to convey information to the user.

[1472] Input: Transmitted audio data

[1473] Output: The audio data played to the user

[1474] This process allows users to report problems in the factory in real time and receive fast, accurate solutions.

[1475] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1476] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1477] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1478] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1479] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1480] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1481] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1482] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1483] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1484] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1485] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1486] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1487] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1488] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1489] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1490] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1491] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1492] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1493] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1494] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1495] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1496] The following is further disclosed regarding the above embodiment.

[1497] (Claim 1)

[1498] means for receiving voice input from a user;

[1499] means for converting the voice input into text data;

[1500] means for transmitting the converted text data to a server;

[1501] means for collecting information based on the received text data;

[1502] A means for generating the collected information as a document;

[1503] a means for converting the generated material into emotive speech;

[1504] means for playing the audio;

[1505] A system including:

[1506] (Claim 2)

[1507] 10. The system of claim 1, which provides information and generates materials in real time based on a user's voice input.

[1508] (Claim 3)

[1509] 2. The system according to claim 1, wherein the generated material is transmitted from the server to a terminal, and the terminal performs an audio replay.

[1510] "Example 1"

[1511] (Claim 1)

[1512] means for receiving voice input from a user;

[1513] means for converting the voice input into text information;

[1514] means for transmitting the converted character information to a data processing device;

[1515] means for collecting data based on the received text information;

[1516] a means for generating a document from the collected data;

[1517] means for converting the generated document into audio data;

[1518] means for reproducing the audio data;

[1519] A system including:

[1520] (Claim 2)

[1521] 10. The system of claim 1, which provides information and generates documents in real time based on a user's voice input.

[1522] (Claim 3)

[1523] 2. The system according to claim 1, wherein the generated document is transmitted from the data processing device to a terminal and is played back as audio by the terminal.

[1524] "Application Example 1"

[1525] (Claim 1)

[1526] means for receiving voice input from a user;

[1527] means for converting the voice input into text data;

[1528] means for transmitting the converted text data to a server;

[1529] means for collecting information based on the received text data;

[1530] A means for generating the collected information as a document;

[1531] a means for converting the generated material into emotive speech;

[1532] means for playing the audio;

[1533] The means available in the autonomous vehicle;

[1534] a means for providing information and generating materials in real time;

[1535] A means for generating materials using a generative AI model;

[1536] means for transmitting and playing audio to a terminal within the autonomous vehicle;

[1537] A system including:

[1538] (Claim 2)

[1539] 10. The system of claim 1, which provides information and generates materials in real time based on a user's voice input.

[1540] (Claim 3)

[1541] The system according to claim 1, wherein the generated material is transmitted from the server to a terminal and played back as audio by the terminal in the autonomous vehicle.

[1542] "Example 2: Combining Emotion Engines"

[1543] (Claim 1)

[1544] means for receiving voice input from a user;

[1545] means for converting the voice input into text data;

[1546] means for transmitting the converted text data to a server;

[1547] means for recognizing a user's emotion based on the received text data;

[1548] a means for collecting information based on the recognized emotion information and text data;

[1549] A means for generating the collected information as a document;

[1550] a means for converting the generated material into emotive speech;

[1551] means for playing the audio;

[1552] A system including:

[1553] (Claim 2)

[1554] 2. The system according to claim 1, wherein the response content and tone of the voice output are adjusted according to the user's emotional information.

[1555] (Claim 3)

[1556] 2. The system according to claim 1, wherein the generated material is transmitted from the server to a terminal, and the terminal performs an audio replay.

[1557] "Application example 2 when combining emotion engines"

[1558] (Claim 1)

[1559] means for receiving voice input from a user;

[1560] means for converting the voice input into text data;

[1561] means for transmitting the converted text data to a server;

[1562] means for collecting information based on the received text data;

[1563] A means for generating the collected information as a document;

[1564] a means for converting the generated material into emotive speech;

[1565] means for adjusting the generated speech to the emotional state of the user;

[1566] means for playing the audio;

[1567] A system including:

[1568] (Claim 2)

[1569] The system of claim 1 receives real-time problem reports, provides solutions to the problems by referencing a historical database, and explains the solutions in an emotive voice.

[1570] (Claim 3)

[1571] 2. The system according to claim 1, wherein the generated materials and audio are transmitted from the server to the terminal, and the audio is replayed by the terminal. [Explanation of symbols]

[1572] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving voice input from a user; means for converting the voice input into text data; means for transmitting the converted text data to a server; means for collecting information based on the received text data; A means for generating the collected information as a document; a means for converting the generated material into emotive speech; means for playing the audio; A system including:

2. The system of claim 1 , which provides information and generates materials in real time based on a user's voice input.

3. 2. The system according to claim 1, wherein the generated material is transmitted from the server to the terminal, and the audio is replayed by the terminal.

Images