System

Abstract

To provide a system for converting voice data into a text with high accuracy, editing it and further converting it into a natural voice.SOLUTION: The system includes means for receiving speech data, means for utilizing a generative model to convert the speech data to text data, means for editing the converted text data, means for saving the edited text data, means for utilizing the generative model to convert the text data to speech data, and means for playing back the converted speech data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional speech recognition technologies have struggled to convert speech into text accurately and efficiently. Furthermore, there has been a lack of means for editing the text data after converting it from speech data to text data. Even technologies that convert text into speech using natural voices have the problem of lacking naturalness and vividness. Therefore, there was a need for a system that provides a series of processes for converting speech data into text with high accuracy, making the text editable, and further converting it into natural-sounding speech. [Means for solving the problem]

[0005] The present invention provides a system including means for receiving voice data, means for utilizing a generative model for converting voice data to text data, means for editing the converted text data, means for saving the edited text data, means for utilizing a generative model for converting text data to voice data, and means for playing back the converted voice data, thereby enabling a user to convert voice data to text data with high accuracy, edit the text data, and finally convert it into natural-sounding voice and play it back.

[0006] "Audio data" refers to data that has been digitized from audio.

[0007] "Text data" refers to digital data that consists of character information such as alphabets and kanji.

[0008] A "generative model" is an artificial intelligence model that uses algorithms trained from large amounts of data to perform specific tasks.

[0009] "Means for receiving audio data" refers to a device or method for capturing audio data, such as a microphone or a file upload function.

[0010] "Means for using a generative model to convert speech data into text data" refers to a method or system for converting speech data into text data using speech recognition technology.

[0011] "Means for editing converted text data" refers to a device or method that allows a user to modify, add, or delete text data.

[0012] The "means for saving edited text data" refers to a device or method for saving data edited by a user for a long period of time.

[0013] "Means for using a generative model to convert text data into speech data" refers to a method or system that uses speech synthesis technology to output text information as speech.

[0014] "Means for playing back converted audio data" refers to a device or method for playing back audio data through speakers, headphones, etc. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[0037] Overall system configuration

[0038] The system mainly consists of the following components:

[0039] 1. Means of receiving audio data

[0040] 2. Generative model for converting audio data into text data

[0041] 3. Means for editing the converted text data

[0042] 4. How to save edited text data

[0043] 5. Generative model for converting text data into speech data

[0044] 6. Means for playing back converted audio data

[0045] The process of converting voice data into text data

[0046] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server stores the converted text data in a database and sends it to the user's device. The user can check the text data on their device and edit it as necessary.

[0047] Specific examples

[0048] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server, which will then use a generative model to convert the speech to text. During this process, the audio data is transcribed with high accuracy and text data is generated. The generated text data is then displayed to the user, allowing them to correct any errors in specific terms or names.

[0049] Processing to edit text data

[0050] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[0051] Specific examples

[0052] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[0053] The process of converting text data into audio data

[0054] If a user wants to convert part or all of the edited text data into audio data, they select the relevant part on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis. The generated audio data is then sent to the user's device and played back.

[0055] Specific examples

[0056] If a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Audio" button. The server sends this text data to a generative model, which generates natural-sounding audio. This audio data is then sent to the device, where the user can press the play button to listen.

[0057] This completes the detailed description of the embodiment of the present invention. This system allows a user to convert voice data into text with high accuracy, edit the text, and output it again as voice data.

[0058] The processing flow will be explained below.

[0059] The process of converting speech to text

[0060] Step 1:

[0061] The user clicks on the upload button on the platform to upload the audio file.

[0062] Step 2:

[0063] The terminal reads the audio file selected by the user and prepares to send the file to the server.

[0064] Step 3:

[0065] The device sends the audio file to the server as an HTTP POST request.

[0066] Step 4:

[0067] The server checks the received audio file and temporarily saves it.

[0068] Step 5:

[0069] The server sends the audio file to the speech recognition generation model and begins the process of converting it into text data.

[0070] Step 6:

[0071] The generative model analyzes the audio data, generates text data, and returns it to the server.

[0072] Step 7:

[0073] The server stores the generated text data in a database and transmits it to the user's terminal.

[0074] Step 8:

[0075] The terminal displays the acquired text data to the user.

[0076] Text editing process

[0077] Step 1:

[0078] The user uses a text editor on the platform to perform editing operations on the text data.

[0079] Step 2:

[0080] The user clicks the save button to save the text data that he has edited.

[0081] Step 3:

[0082] The terminal prepares to send the edited text data to the server.

[0083] Step 4:

[0084] The device sends the edited text data to the server as an HTTP POST request.

[0085] Step 5:

[0086] The server checks the received edited text data and saves it in the database.

[0087] Text-to-speech processing

[0088] Step 1:

[0089] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[0090] Step 2:

[0091] The terminal prepares to send the selected text data to the server.

[0092] Step 3:

[0093] The terminal sends the selected text data to the server as an HTTP POST request.

[0094] Step 4:

[0095] The server checks the received text data and sends it to the speech synthesis generation model.

[0096] Step 5:

[0097] The generative model analyzes the text data, generates audio data, and returns it to the server.

[0098] Step 6:

[0099] The server transmits the generated voice data to the user's terminal.

[0100] Step 7:

[0101] The terminal prepares to play the received audio data.

[0102] Step 8:

[0103] The terminal plays the audio data to the user.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] There is a need for a system that can convert speech data into text data and then convert that text data back into speech data with high accuracy and naturalness, while also being easy for users to operate. However, conventional technologies have not been able to perform this process efficiently or accurately, and have not yet achieved user satisfaction. Furthermore, achieving real-time conversion and natural speech synthesis has also been a challenge.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes means for converting voice data into text data using a generative AI model, means for saving the edited text data, and means for converting the edited text data into voice data using a generative AI model, thereby enabling the generation of highly accurate text data from voice data, editing of the text data, and reconversion into natural voice data.

[0109] "Audio File" means a digital file containing audio data that a User uploads to the System.

[0110] A "terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[0111] A "server" is a central computer system that processes voice data, stores text data, and synthesizes voice.

[0112] A "generative AI model" is a machine learning model that uses speech recognition technology to convert voice data into text data and speech synthesis technology to convert text data into voice data.

[0113] "Text data" refers to character string data converted from voice data by a generative AI model.

[0114] "Editing" refers to an operation in which a user corrects or adds to text data using a text editor on a terminal.

[0115] "Saving" refers to the process of storing edited text data in a storage device such as a database.

[0116] "Voice data" means audio data converted from text data using a generative AI model.

[0117] "Playback" refers to the operation of outputting the generated audio data as audio again on the terminal.

[0118] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[0119] Overall system configuration

[0120] The system mainly consists of the following elements:

[0121] 1. A way for users to upload audio files

[0122] 2. How the device sends the audio file to the server

[0123] 3. A means for the server to convert voice data into text data using a generative AI model (e.g., speech recognition API or speech synthesis API).

[0124] 4. A means for users to view and edit text data

[0125] 5. Means by which the device sends edited text data to the server

[0126] 6. How the server stores the edited text data

[0127] 7. A means for the server to convert edited text data into voice data using a generative AI model

[0128] 8. Means for the device to play back the generated audio data

[0129] Hardware and software used

[0130] Terminal: A device such as a computer, smartphone, or tablet that is operated by a user.

[0131] Server: A central processing unit for high-performance computing

[0132] Generative AI models: speech recognition API, speech synthesis API (e.g. Google Cloud Speech-to-Text, Amazon Polly)

[0133] Text editor: Software that allows users to edit text data (e.g., Notepad++, Visual Studio Code)

[0134] Database: A database system for storing text data (e.g., MySQL, PostgreSQL)

[0135] Specific examples of processing

[0136] For example, if a user uploads a one-hour meeting recording to the platform, the process goes like this:

[0137] 1. The user uploads the "Meeting Recording.wav" file from their device. Click the upload button on the screen and select the desired audio file from the file selection dialog.

[0138] 2. The device then sends the received audio file to the server using a secure communication protocol (e.g., HTTPS).

[0139] 3. The server passes the voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to generate highly accurate text data.

[0140] 4. The generated text data is stored on the server, and the converted results are sent to the user's device.

[0141] 5. The user checks the text data on the device and corrects it as necessary using a text editor (e.g., Notepad++ or Visual Studio Code). Correct any errors in specific terms or names.

[0142] 6. Once edited, the text data is sent back to the server from the device.

[0143] 7. The server saves the edited text data and passes it to a speech synthesis API (e.g., Amazon Polly) to convert it into natural-sounding voice data.

[0144] 8. The generated audio data is sent to the terminal, and the user can listen to it by pressing the play button.

[0145] Examples of prompt statements

[0146] Prompt to convert voice data to text data:

[0147] "Please convert the following audio data to text data.\nAudio file path: / path / to / meeting_audio.wav"

[0148] Prompt to save edited text data:

[0149] "Please save the following text data.\nCorrected text data:..."

[0150] Prompt to convert text data to speech data:

[0151] "Please convert the following text data into audio data.\nText data:..."

[0152] This allows text data to be generated from voice data with high accuracy, the text data to be edited by the user, and the text data to be output again as voice data.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] The user uploads an audio file. The user selects an audio file from the file selection dialog on the device and clicks the upload button. This causes the device to obtain the path of the selected audio file and confirm that the file is ready.

[0156] Input: Audio file

[0157] Output: Path to the audio file

[0158] Step 2:

[0159] The device sends the audio file to the server. The device reads the audio file based on the file path and sends an HTTP request to the server using the HTTPS protocol. The request includes the binary data of the audio file.

[0160] Input: Path to the audio file

[0161] Output: None (transmission process)

[0162] Step 3:

[0163] The server converts the received audio file into text data using a generative AI model. The server then passes the audio data to a speech recognition API (e.g., Google Cloud Speech-to-Text) for highly accurate transcription. The generated text data is stored in a database.

[0164] Input: Audio file

[0165] Output: Text data

[0166] Step 4:

[0167] The server sends the converted text data to the device, which then generates an HTTP response and returns the transcribed text data in JSON format to the device.

[0168] Input: Text data

[0169] Output: Text data (sent to the terminal)

[0170] Step 5:

[0171] The user checks and edits the text data on the terminal. The user opens the text data using a specific text editor (e.g., Notepad++, Visual Studio Code), makes necessary corrections (e.g., correcting typos or replacing terms), and clicks the save button when editing is complete.

[0172] Input: Text data

[0173] Output: Edited text data

[0174] Step 6:

[0175] The device collects the edited text data based on the user's operations, generates JSON format data, and sends it to the server using the HTTPS protocol.

[0176] Input: Edited text data

[0177] Output: None (transmission process)

[0178] Step 7:

[0179] The server saves the edited text data in a database. The server analyzes the received text data and stores it in the database. After saving is complete, the server notifies the terminal of the status.

[0180] Input: Edited text data

[0181] Output: Save successful status

[0182] Step 8:

[0183] To convert all or part of the edited text data into audio data, the user selects the relevant part on the device and clicks the "Convert to audio" button. The device then sends the selected text data to the server.

[0184] Input: Selected text data

[0185] Output: None (transmission process)

[0186] Step 9:

[0187] The server converts the received text data into voice data using a generative AI model. The server then passes the text data to a speech synthesis API (e.g., Amazon Polly) to generate natural-sounding voice data. The generated voice data is then returned to the device in JSON format.

[0188] Input: Text data

[0189] Output: Audio data

[0190] Step 10:

[0191] The device plays the generated audio data. The device analyzes the audio data received from the server and plays it using the built-in audio player. The user can press the play button to listen to the generated audio.

[0192] Input: Audio data

[0193] Output: None (playback processing)

[0194] (Application example 1)

[0195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] Conventional factory robots lack the ability to accurately convert voice instructions into text in real time and provide voice feedback based on the results of text analysis. This makes it difficult for them to communicate smoothly with operators and efficiently execute tasks. The purpose of this invention is to solve this problem and improve productivity and communication within factories.

[0197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0198] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and means for analyzing the text data and executing a task, which enables the factory robot to convert voice instructions into text in real time and quickly provide the analysis results as voice feedback to the operator.

[0199] "Voice data" refers to the recording, transmission and processing of human voice as electronic signals.

[0200] "Text data" refers to character string information converted from voice data, or digital character information generated based on that.

[0201] A "generative model" is an algorithm that uses machine learning and artificial intelligence techniques to convert voice data into text data and vice versa.

[0202] "Real-time" is a time concept that means that instructions and data are processed and reflected almost immediately.

[0203] "Analysis" refers to the data processing method used to understand and process the generated text data and take appropriate action or feedback.

[0204] "Feedback" means that the system returns analysis results to the operator in the form of audio data or the like.

[0205] "Speech synthesis" is a technology that converts text data into natural-sounding speech.

[0206] A "server" is a computer system that centrally receives, processes, converts, stores, and transmits audio data.

[0207] An "operator" is a human operator who gives voice instructions to and receives feedback from a factory robot.

[0208] MODE FOR CARRYING OUT THE INVENTION

[0209] Overall system configuration

[0210] This invention is applied to a robot system in a factory, and is a system that converts voice instructions into text data in real time and enables the analysis results to be fed back as voice data. This system mainly consists of the following elements:

[0211] 1. Means of receiving audio data

[0212] This uses the factory robot's microphone to receive voice instructions from the operator.

[0213] 2. Using generative models to convert audio data into text

[0214] Use a speech recognition library (e.g. speech_recognition) to convert the received audio into text.

[0215] 3. A means of analyzing text data and performing tasks

[0216] The converted text data is analyzed and the robot performs the specified task.

[0217] 4. A means of outputting the results of text data analysis as an audio report

[0218] Using a speech synthesis library (e.g. gTTS), the analysis results are generated as natural-sounding speech and reported to the operator.

[0219] This allows operators to give instructions to the robot in real time, enabling it to perform tasks quickly and accurately.

[0220] Hardware and software used

[0221] Hardware:

[0222] Microphone: Collects the operator's voice instructions.

[0223] Speaker: Plays audio feedback.

[0224] software:

[0225] Speech Recognition Library (speech_recognition): Collects voice data and converts it into text data.

[0226] Text-to-Speech Library (gTTS): Used to convert text data into speech data and play the generated speech data.

[0227] Data processing and calculation

[0228] Acquisition and transcription of audio data:

[0229] The microphone receives voice instructions given by the user and transmits them to the server, where the speech recognition library converts the received voice data into text data.

[0230] Analyzing text data and performing tasks:

[0231] The server analyzes the converted text data and determines the corresponding actions for the factory robot to carry out the instructions.

[0232] Audio data generation and feedback:

[0233] Feedback information such as task completion and error messages is generated as text data and converted to voice data using a speech synthesis library. The generated voice data is reported to the operator through a speaker.

[0234] Specific examples

[0235] If the operator gives you the following voice prompts:

[0236] "Please stop the conveyor belt on line 1."

[0237] 1. Speech Recognition:

[0238] The microphone receives the voice and converts it into text data such as "Please stop the conveyor belt on line 1" through a speech recognition library.

[0239] 2. Analysis and execution:

[0240] The server analyzes the text data and instructs the robot to stop the conveyor belt.

[0241] 3. Audio feedback:

[0242] As a result of the analysis, the robot reports to the operator "The conveyor belt has stopped" using voice data generated by a speech synthesis library.

[0243] Prompt Sentence Examples

[0244] Example of an input prompt for a generative AI model:

[0245] Please show me how to convert the following voice instructions into text data and convert the analysis results back into voice:

[0246] Voice: "Please stop the conveyor belt on line 1"

[0247] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0248] Step 1:

[0249] Receiving and sending audio data

[0250] The user acts as an operator and issues instructions into the microphone. The microphone receives the voice data, and the terminal transmits the voice data to the server.

[0251] Specific behavior:

[0252] Input: User's voice data

[0253] Data processing: Capture audio with a microphone and convert it into a digital signal

[0254] Output: Audio data sent to the server

[0255] Step 2:

[0256] Converting audio data to text

[0257] The server converts the received voice data into text data using a voice recognition library.

[0258] Specific behavior:

[0259] Input: Audio data

[0260] Data calculation: Analyzes speech using a speech recognition algorithm (speech_recognition) and converts it into text

[0261] Output: Text data

[0262] Step 3:

[0263] Text data analysis and task instructions

[0264] The server analyzes the converted text data and instructs the robot on the appropriate task.

[0265] Specific behavior:

[0266] Input: Text data

[0267] Data Computing: Using text analysis algorithms to interpret instructions and determine the tasks to be performed

[0268] Output: Instructions to the robot to execute the task

[0269] Step 4:

[0270] Executing a task

[0271] The robot performs actual operations based on task instructions received from the server, such as stopping a conveyor belt.

[0272] Specific behavior:

[0273] Input: Task execution instructions

[0274] Data processing: The robot performs physical actions according to instructions

[0275] Output: Task completion (e.g. conveyor belt stopped)

[0276] Step 5:

[0277] Generate feedback on task completion

[0278] The server confirms the completion of the task and generates the details as text data.

[0279] Specific behavior:

[0280] Input: Task execution result

[0281] Data processing: Convert task completion information into text data

[0282] Output: Text data (e.g. "Conveyor belt has stopped")

[0283] Step 6:

[0284] Text-to-speech conversion

[0285] The text data generated by the server is converted into voice data using a voice synthesis library.

[0286] Specific behavior:

[0287] Input: Text data

[0288] Data Computing: Convert text to speech using a text-to-speech algorithm (gTTS)

[0289] Output: Audio data

[0290] Step 7:

[0291] Playback and reporting of audio data

[0292] The server sends the converted voice data to the terminal and reports it to the user through the speaker.

[0293] Specific behavior:

[0294] Input: Audio data

[0295] Data calculation: The device plays the audio data received on the speaker.

[0296] Output: User confirms audio feedback

[0297] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0298] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide users with highly accurate and natural interactions.

[0299] Overall system configuration

[0300] The system mainly consists of the following components:

[0301] 1. Means of receiving audio data

[0302] 2. Generative model for converting audio data into text data

[0303] 3. Means for editing the converted text data

[0304] 4. How to save edited text data

[0305] 5. Generative model for converting text data into speech data

[0306] 6. Means for playing back converted audio data

[0307] 7. Emotion engine for recognizing user emotions

[0308] The process of converting voice data into text data

[0309] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server then sends the text data to an emotion engine to analyze the user's emotions. Text data tagged with emotion-based tags is generated, stored in a database, and sent to the user's device. The user can review the text data on their device and edit it as necessary.

[0310] Specific examples

[0311] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server. The server will then use a generative model to convert the speech into text. During this process, the audio data is transcribed with high accuracy and text data is generated. The emotion engine will then analyze the user's emotions and assign emotion tags to them. The generated text data will then be displayed to the user, allowing them to correct any errors in specific terms or names.

[0312] Processing to edit text data

[0313] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[0314] Specific examples

[0315] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[0316] The process of converting text data into audio data

[0317] If a user wants to convert part or all of the edited text data into voice data, they select the relevant section on their device and click the "Convert to voice" button. The device then sends the selected text data to the server, which then sends the data to the generative model for voice synthesis. The generated voice data is generated by the emotion engine in a way that reflects the user's emotions, and is then sent to the user's device and played back.

[0318] Specific examples

[0319] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[0320] This concludes the detailed description of the embodiment of the present invention. This system allows users to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[0321] The processing flow will be explained below.

[0322] Converting speech to text and performing sentiment analysis

[0323] Step 1:

[0324] The user clicks on the upload button on the platform to upload the audio file.

[0325] Step 2:

[0326] The device reads the audio file selected by the user and prepares it for transmission to the server.

[0327] Step 3:

[0328] The device sends the audio file to the server as an HTTP POST request.

[0329] Step 4:

[0330] The server checks the received audio file and temporarily saves it.

[0331] Step 5:

[0332] The server sends the audio file to the speech recognition generation model, which starts the process of converting the audio data into text data.

[0333] Step 6:

[0334] The generative model analyzes the audio data, generates text data, and returns it to the server.

[0335] Step 7:

[0336] The server sends the generated text data to the emotion engine.

[0337] Step 8:

[0338] The emotion engine analyzes the text data and tags it with the user's emotions.

[0339] Step 9:

[0340] The server stores emotion-tagged text data in a database and sends it to the user's device.

[0341] Step 10:

[0342] The terminal displays the acquired text data to the user.

[0343] Text editing process

[0344] Step 1:

[0345] The user uses a text editor on the platform to perform editing operations on the text data.

[0346] Step 2:

[0347] The user clicks the save button to save the text data that he has edited.

[0348] Step 3:

[0349] The terminal prepares to send the edited text data to the server.

[0350] Step 4:

[0351] The device sends the edited text data to the server as an HTTP POST request.

[0352] Step 5:

[0353] The server checks the received edited text data and saves it in the database.

[0354] Converting text to speech and reflecting emotions

[0355] Step 1:

[0356] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[0357] Step 2:

[0358] The terminal prepares to send the selected text data to the server.

[0359] Step 3:

[0360] The terminal sends the selected text data to the server as an HTTP POST request.

[0361] Step 4:

[0362] The server checks the received text data and sends it to the speech synthesis generation model.

[0363] Step 5:

[0364] The generative model analyzes the text data, generates audio data, and returns it to the server.

[0365] Step 6:

[0366] The server sends the generated voice data to the emotion engine, where it processes the data to reflect the emotion.

[0367] Step 7:

[0368] The emotion engine adds emotion information to the voice data, and the final voice data is sent to the server.

[0369] Step 8:

[0370] The server transmits the voice data that reflects the emotion to the user's terminal.

[0371] Step 9:

[0372] The device prepares to play the received audio data.

[0373] Step 10:

[0374] The terminal plays the audio data to the user.

[0375] Example 2

[0376] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0377] Conventional systems that convert voice data into text data have limitations in the accuracy of voice recognition and the naturalness of synthesized voice, and one problem is that it is particularly difficult to reflect the user's emotions. This makes it difficult for users to communicate naturally, including their emotions. The present invention aims to solve these problems and realize highly accurate conversion of voice data into text and generation of voice data that reflects emotions.

[0378] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0379] In this invention, the server includes means for receiving voice data, means for using a generative model to convert voice data into text data, means for performing emotion recognition on the converted text data, means for saving emotion-tagged text data, means for saving edited text data, means for using a generative model to convert text data into voice data, and means for playing back the converted voice data. This enables a user to convert voice data into text, edit text data, and generate voice data that reflects emotions with high accuracy and in a natural way.

[0380] "Audio data" refers to data in which audio is recorded in digital format.

[0381] "Text data" refers to data obtained by converting voice data into character information.

[0382] A "generative model" is a machine learning model used to convert voice data to text data or vice versa.

[0383] "Emotion recognition" is a technology that analyzes and identifies a user's emotions from text data and voice data.

[0384] An "emotion tag" is a tag that includes information about an emotion that is assigned to text data by emotion recognition.

[0385] A "server" is a computer system that receives voice data, converts it, recognizes emotions, stores it, and transmits the data.

[0386] A "terminal" is a device that is operated by a user and is used to send and receive voice data and text data.

[0387] "Real time" refers to a processing format in which the process from receiving voice data to generating text data is carried out without delay.

[0388] "Speech synthesis" is a technology that generates synthetic speech based on text data.

[0389] The present invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interaction. The specific operation of this system is as follows.

[0390] Hardware and software used

[0391] Hardware

[0392] User's input device (terminal): PC, smartphone, tablet, etc.

[0393] Server: High-performance cloud or on-premise server

[0394] software

[0395] Speech recognition API: The general name is "Speech recognition API"

[0396] Emotion recognition engine: The generic name "emotion recognition engine"

[0397] Speech synthesis API: The general name is "Speech synthesis API"

[0398] Database: The general term "database software"

[0399] Text editor: A common text editing software installed on the device.

[0400] Overall system configuration

[0401] The system mainly consists of the following components:

[0402] 1. Means of receiving audio data

[0403] 2. Generative model for converting audio data into text data

[0404] 3. Means for editing the converted text data

[0405] 4. Means of Emotion Recognition

[0406] 5. A means of storing emotion-tagged text data

[0407] 6. How to save edited text data

[0408] 7. Generative model for converting text data into speech data

[0409] 8. Means for playing back converted audio data

[0410] Example of operation

[0411] The process of converting voice data into text data

[0412] When a user uploads an audio file, the device sends the file to the server. The server converts the audio data into text using a speech recognition API, and then sends the text to an emotion recognition engine for emotional analysis. Emotion-based tagged text data is generated, stored in a database, and sent to the user's device. The user can then review the text data on their device and edit it as necessary.

[0413] Specific prompt examples:

[0414] User: Upload a 1-hour meeting recording

[0415] Terminal: Send a file to the server

[0416] Server: Convert speech to text using speech recognition API

[0417] Server: Analyze emotions using an emotion recognition engine

[0418] Server: Generate emotion-tagged text data and store it in a database

[0419] Terminal: displays text data to the user

[0420] Processing to edit text data

[0421] The user can edit the text data using a text editor on the terminal. After editing, the text data is sent to the server at the user's instruction and saved in the database again.

[0422] Specific prompt examples:

[0423] User: Open text data in a text editor

[0424] User: Correct typos and omissions

[0425] User: Clicks the Save button to send the changes to the server

[0426] Server: Save the modified text data to the database

[0427] The process of converting text data into audio data

[0428] If a user wants to convert the edited text data into audio data, they select the relevant section on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then uses a speech synthesis API to generate audio data. This audio data is generated using an emotion recognition engine to reflect the user's emotions, and is then sent to the user's device and played back.

[0429] Specific prompt examples:

[0430] User: Select part of edited text data

[0431] User: Clicks the "Convert to Audio" button

[0432] Server: Sends the selected text data to the speech synthesis API

[0433] Server: Generates natural voice using emotion recognition engine

[0434] Server: Sends the generated audio data to the device

[0435] User: Press the play button to check

[0436] The above is a specific embodiment of the program processing of this system. This system allows the user to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[0437] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0438] Step 1: Receiving audio data

[0439] The user selects an audio file on the terminal and clicks the "Upload" button. The input is the audio file selected by the user. The terminal sends this audio file to the server. The output is the audio file sent to the server.

[0440] Step 2: Save the audio data to the server

[0441] The server saves the received audio file in a specific folder. The received audio file is the input, and the saved audio file is the output. At this time, the terminal displays the file transmission status on the screen.

[0442] Step 3: Converting audio data to text data

[0443] The server sends the saved voice data to a voice recognition API (commonly known as a "voice recognition API"). The voice data is input, and the API analyzes it to generate text data. The text data is obtained as output. Specifically, the API returns the text data "We discussed this project."

[0444] Step 4: Emotion Recognition

[0445] The server sends the generated text data to an emotion recognition engine (commonly known as an "emotion recognition engine"). The engine receives text data as input and identifies the user's emotion from the text data. The output is text data tagged with an emotion. Specifically, the emotion engine assigns a tag of "joy."

[0446] Step 5: Saving emotion-tagged text data

[0447] The server stores emotion-tagged text data in a database.,The input is emotion-tagged text data, and the output is the,text data stored in the database.

[0448] Step 6: Sending text data to the user

[0449] The server sends emotion-tagged text data stored in the database to the user's device. The input is text data from the database, and the output is text data sent to the user's device. The user can view this data.

[0450] Step 7: Editing text data

[0451] The user opens the received text data using a text editor on the terminal and makes the necessary corrections. The input is the transmitted text data, and the user performs editing tasks such as correcting typos and omissions. The edited text data is obtained as output. In concrete terms, the user changes "project" to "plan."

[0452] Step 8: Save the edited text data to the server

[0453] The user completes editing the text and clicks the "Save" button. The terminal sends the edited text data to the server. The input is the edited text data, and the server saves it in the database. The output is the corrected text data saved in the database.

[0454] Step 9: Converting text data to audio data

[0455] The user selects the edited text data and clicks the "Convert to speech" button. The device sends the selected text data to the server. The selected text data is input, and the server sends it to a speech synthesis API (commonly known as a "speech synthesis API") to generate speech data. The generated speech data is obtained as output.

[0456] Step 10: Send and play the generated audio data to the user

[0457] The server sends the generated voice data to the user's device. The generated voice data is input, and the voice data sent to the device is obtained as output. The user presses the play button to check the voice. Specifically, the user clicks the play button, and the text is played back in a natural voice.

[0458] (Application example 2)

[0459] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0460] Conventional food delivery services have problems that can easily lead to dissatisfaction due to misreading of order details and lack of emotional response. In particular, when ordering by voice, when users provide emotional feedback, the emotional information is not properly conveyed, making it difficult to improve service quality. Therefore, a system is needed that can accurately convert users' voice input into text and appropriately reflect emotional information, thereby deepening mutual understanding between service providers and users and improving the quality of customer support.

[0461] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0462] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and an emotion engine for recognizing a user's emotion and reflecting the emotion information in the text data and voice data. This allows the user's voice input to be converted into text data with high accuracy, and makes it possible to appropriately notify the service provider of the user's emotion information.

[0463] 1. "Voice Data" means a digitized recording of a user's speech.

[0464] 2. "Text data" means digital information that has been converted from audio data into textual information.

[0465] 3. A "generative model" is a model that has been trained using machine learning or AI techniques to perform a specific task.

[0466] 4. "Means" are the methods of equipment or software used to achieve a particular purpose.

[0467] 5. "Emotion engine" is a technology that analyzes a user's voice and text data to recognize and tag emotions.

[0468] 6. "Customer Support Team" refers to the department within the service provider that handles user inquiries and resolves problems.

[0469] 7. "Notice" means a communication that conveys information to a specific recipient.

[0470] 8. "Service Provider" means a company or organization that provides food delivery or other services to Users.

[0471] 9. "Real time" means that input and output occur immediately without delay.

[0472] 10. "Speech synthesis function" refers to technology that converts text data into voice data and plays it back as natural-sounding voice.

[0473] MODE FOR CARRYING OUT THE INVENTION

[0474] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. The system integrates voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interactions.

[0475] Overall system configuration

[0476] The system mainly consists of the following components:

[0477] 1. Means of receiving audio data

[0478] 2. Using generative models to convert audio data into text

[0479] 3. Means for editing the converted text data

[0480] 4. How to save edited text data

[0481] 5. Using generative models to convert text data into audio data

[0482] 6. Means for playing back converted audio data

[0483] 7. An emotion engine that recognizes user emotions and reflects that emotional information in text and voice data.

[0484] 8. Means for recognizing emotions based on user voice input and sending notifications containing emotion information to service providers

[0485] The process of converting voice data into text data

[0486] When a user speaks, the device sends this voice data to the server. The server then sends the voice data to a generative model for speech recognition. The generated text data is analyzed by an emotion engine to analyze the user's emotional information, and an emotion tag is added to the text data. This text data is then stored in a database and sent to the user's device.

[0487] Specific examples

[0488] For example, if a user reports by voice that "my order is late and I'm frustrated," this voice data is sent by the device to the server. The server uses a generative model to convert the speech into text, and in the process, the emotion engine recognizes the emotion "frustrated." Then, emotion-tagged text data is generated, stored in a database, and sent to the user's device.

[0489] Processing to edit text data

[0490] When a user edits text data, the user can make corrections using a text editor on the device. Once edited, the text data is sent to the server at the user's instruction and saved in the database again.

[0491] Specific examples

[0492] The user can check the generated text data and correct any typos, omissions, or translation errors. For example, it is possible to change "The order will not arrive quickly" to "The order is delayed." Once the corrections are complete, the user clicks the save button and sends the corrections to the server.

[0493] The process of converting text data into audio data

[0494] If a user wants to convert part or all of the edited text data into speech data, they select the relevant part on their device and click the "Convert to speech" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis.

[0495] Specific examples

[0496] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[0497] Hardware and software used

[0498] The hardware used is a cloud server (e.g., Amazon EC2) for high-speed processing and managing large amounts of data, and a smartphone for user interaction. The software used is a generative model for speech recognition and emotion recognition using PyTorch and Transformers.

[0499] Prompt Sentence Examples

[0500] If a user says, "I'd like to order a pizza and salad," create a program that takes the voice data, converts it to text, recognizes the sentiment, and notifies your customer support team.

[0501] As described above, this system can convert voice data into text with high accuracy, add emotional information, and output it as voice data, thereby improving the quality of customer support for food delivery services.

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1:

[0504] The user inputs voice orders or inquiries into the terminal. The input voice data is digitized by the terminal and sent to the server, where it is captured into the system.

[0505] Input: User's voice data

[0506] Output: The digitized audio data is sent to the server.

[0507] Step 2:

[0508] The server sends the received speech data to a generative model (speech recognition model) to convert the speech into text data. High-accuracy speech recognition is achieved using models such as PyTorch and Wav2Vec2. The generated text data is then sent to the next processing step.

[0509] Input: Digitized audio data

[0510] Output: Converted text data

[0511] Step 3:

[0512] The server sends the converted text data to the emotion engine for emotion analysis. The emotion engine analyzes the text data, recognizes the user's emotional information, and tags it. The text data with the added emotional information is stored in a database and sent to the user's device.

[0513] Input: Converted text data

[0514] Output: Text data with emotional information added

[0515] Step 4:

[0516] The user can use a text editor on the device to check the generated text data and edit it as necessary. After editing, the text data is sent from the device to the server and saved in the database.

[0517] Input: Text data with emotional information added

[0518] Output: Edited text data

[0519] Step 5:

[0520] When a user wants to convert edited text data into voice data, they select the relevant part on their device and issue a command to convert it into voice. The device then sends the selected text data to the server, which then generates voice data using a generative model (voice synthesis model).

[0521] Input: Edited text data

[0522] Output: Generated audio data

[0523] Step 6:

[0524] The generated voice data is generated in a way that reflects the user's emotional information through the emotion engine. The voice data is sent to the device, and the user can listen to the voice data by pressing the play button.

[0525] Input: Generated audio data

[0526] Output: Audio data sent to the user's device

[0527] Step 7:

[0528] Based on the user's voice input and edits, a means for sending notifications containing emotional information to the service provider is activated, for example, a notification such as "The user is unhappy with the order" is sent to the customer support team in real time.

[0529] Input: Voice input or edited data

[0530] Output: A notification containing emotion information is sent to the service provider.

[0531] The above are the specific processing steps of the system of the present invention. This configuration enables the user's voice input to be converted into text with high accuracy, emotional information to be appropriately reflected, and the quality of customer support to be improved.

[0532] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0533] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0534] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0535] [Second embodiment]

[0536] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0537] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0538] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0539] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0540] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0541] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0542] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0543] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0544] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0545] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0546] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0547] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0548] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[0549] Overall system configuration

[0550] The system mainly consists of the following components:

[0551] 1. Means of receiving audio data

[0552] 2. Generative model for converting audio data into text data

[0553] 3. Means for editing the converted text data

[0554] 4. How to save edited text data

[0555] 5. Generative model for converting text data into speech data

[0556] 6. Means for playing back converted audio data

[0557] The process of converting voice data into text data

[0558] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server stores the converted text data in a database and sends it to the user's device. The user can check the text data on their device and edit it as necessary.

[0559] Specific examples

[0560] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server, which will then use a generative model to convert the speech to text. During this process, the audio data is transcribed with high accuracy and text data is generated. The generated text data is then displayed to the user, allowing them to correct any errors in specific terms or names.

[0561] Processing to edit text data

[0562] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[0563] Specific examples

[0564] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[0565] The process of converting text data into audio data

[0566] If a user wants to convert part or all of the edited text data into audio data, they select the relevant part on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis. The generated audio data is then sent to the user's device and played back.

[0567] Specific examples

[0568] If a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Audio" button. The server sends this text data to a generative model, which generates natural-sounding audio. This audio data is then sent to the device, where the user can press the play button to listen.

[0569] This completes the detailed description of the embodiment of the present invention. This system allows a user to convert voice data into text with high accuracy, edit the text, and output it again as voice data.

[0570] The processing flow will be explained below.

[0571] The process of converting speech to text

[0572] Step 1:

[0573] The user clicks on the upload button on the platform to upload the audio file.

[0574] Step 2:

[0575] The terminal reads the audio file selected by the user and prepares to send the file to the server.

[0576] Step 3:

[0577] The device sends the audio file to the server as an HTTP POST request.

[0578] Step 4:

[0579] The server checks the received audio file and temporarily saves it.

[0580] Step 5:

[0581] The server sends the audio file to the speech recognition generation model and begins the process of converting it into text data.

[0582] Step 6:

[0583] The generative model analyzes the audio data, generates text data, and returns it to the server.

[0584] Step 7:

[0585] The server stores the generated text data in a database and transmits it to the user's terminal.

[0586] Step 8:

[0587] The terminal displays the acquired text data to the user.

[0588] Text editing process

[0589] Step 1:

[0590] The user uses a text editor on the platform to perform editing operations on the text data.

[0591] Step 2:

[0592] The user clicks the save button to save the text data that he has edited.

[0593] Step 3:

[0594] The terminal prepares to send the edited text data to the server.

[0595] Step 4:

[0596] The device sends the edited text data to the server as an HTTP POST request.

[0597] Step 5:

[0598] The server checks the received edited text data and saves it in the database.

[0599] Text-to-speech processing

[0600] Step 1:

[0601] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[0602] Step 2:

[0603] The terminal prepares to send the selected text data to the server.

[0604] Step 3:

[0605] The terminal sends the selected text data to the server as an HTTP POST request.

[0606] Step 4:

[0607] The server checks the received text data and sends it to the speech synthesis generation model.

[0608] Step 5:

[0609] The generative model analyzes the text data, generates audio data, and returns it to the server.

[0610] Step 6:

[0611] The server transmits the generated voice data to the user's terminal.

[0612] Step 7:

[0613] The terminal prepares to play the received audio data.

[0614] Step 8:

[0615] The terminal plays the audio data to the user.

[0616] Example 1

[0617] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] There is a need for a system that can convert speech data into text data and then convert that text data back into speech data with high accuracy and naturalness, while also being easy for users to operate. However, conventional technologies have not been able to perform this process efficiently or accurately, and have not yet achieved user satisfaction. Furthermore, achieving real-time conversion and natural speech synthesis has also been a challenge.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0620] In this invention, the server includes means for converting voice data into text data using a generative AI model, means for saving the edited text data, and means for converting the edited text data into voice data using a generative AI model, thereby enabling the generation of highly accurate text data from voice data, editing of the text data, and reconversion into natural voice data.

[0621] "Audio File" means a digital file containing audio data that a User uploads to the System.

[0622] A "terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[0623] A "server" is a central computer system that processes voice data, stores text data, and synthesizes voice.

[0624] A "generative AI model" is a machine learning model that uses speech recognition technology to convert voice data into text data and speech synthesis technology to convert text data into voice data.

[0625] "Text data" refers to character string data converted from voice data by a generative AI model.

[0626] "Editing" refers to an operation in which a user corrects or adds to text data using a text editor on a terminal.

[0627] "Saving" refers to the process of storing edited text data in a storage device such as a database.

[0628] "Voice data" means audio data converted from text data using a generative AI model.

[0629] "Playback" refers to the operation of outputting the generated audio data as audio again on the terminal.

[0630] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[0631] Overall system configuration

[0632] The system mainly consists of the following elements:

[0633] 1. A way for users to upload audio files

[0634] 2. How the device sends the audio file to the server

[0635] 3. A means for the server to convert voice data into text data using a generative AI model (e.g., speech recognition API or speech synthesis API).

[0636] 4. A means for users to view and edit text data

[0637] 5. Means by which the device sends edited text data to the server

[0638] 6. How the server stores the edited text data

[0639] 7. A means for the server to convert edited text data into voice data using a generative AI model

[0640] 8. Means for the device to play back the generated audio data

[0641] Hardware and software used

[0642] Terminal: A device such as a computer, smartphone, or tablet that is operated by a user.

[0643] Server: A central processing unit for high-performance computing

[0644] Generative AI models: speech recognition API, speech synthesis API (e.g. Google Cloud Speech-to-Text, Amazon Polly)

[0645] Text editor: Software that allows users to edit text data (e.g., Notepad++, Visual Studio Code)

[0646] Database: A database system for storing text data (e.g., MySQL, PostgreSQL)

[0647] Specific examples of processing

[0648] For example, if a user uploads a one-hour meeting recording to the platform, the process goes like this:

[0649] 1. The user uploads the "Meeting Recording.wav" file from their device. Click the upload button on the screen and select the desired audio file from the file selection dialog.

[0650] 2. The device then sends the received audio file to the server using a secure communication protocol (e.g., HTTPS).

[0651] 3. The server passes the voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to generate highly accurate text data.

[0652] 4. The generated text data is stored on the server, and the converted results are sent to the user's device.

[0653] 5. The user checks the text data on the device and corrects it as necessary using a text editor (e.g., Notepad++ or Visual Studio Code). Correct any errors in specific terms or names.

[0654] 6. Once edited, the text data is sent back to the server from the device.

[0655] 7. The server saves the edited text data and passes it to a speech synthesis API (e.g., Amazon Polly) to convert it into natural-sounding voice data.

[0656] 8. The generated audio data is sent to the terminal, and the user can listen to it by pressing the play button.

[0657] Examples of prompt statements

[0658] Prompt to convert voice data to text data:

[0659] "Please convert the following audio data to text data.\nAudio file path: / path / to / meeting_audio.wav"

[0660] Prompt to save edited text data:

[0661] "Please save the following text data.\nCorrected text data:..."

[0662] Prompt to convert text data to speech data:

[0663] "Please convert the following text data into audio data.\nText data:..."

[0664] This allows text data to be generated from voice data with high accuracy, the text data to be edited by the user, and the text data to be output again as voice data.

[0665] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0666] Step 1:

[0667] The user uploads an audio file. The user selects an audio file from the file selection dialog on the device and clicks the upload button. This causes the device to obtain the path of the selected audio file and confirm that the file is ready.

[0668] Input: Audio file

[0669] Output: Path to the audio file

[0670] Step 2:

[0671] The device sends the audio file to the server. The device reads the audio file based on the file path and sends an HTTP request to the server using the HTTPS protocol. The request includes the binary data of the audio file.

[0672] Input: Path to the audio file

[0673] Output: None (transmission process)

[0674] Step 3:

[0675] The server converts the received audio file into text data using a generative AI model. The server then passes the audio data to a speech recognition API (e.g., Google Cloud Speech-to-Text) for highly accurate transcription. The generated text data is stored in a database.

[0676] Input: Audio file

[0677] Output: Text data

[0678] Step 4:

[0679] The server sends the converted text data to the device, which then generates an HTTP response and returns the transcribed text data in JSON format to the device.

[0680] Input: Text data

[0681] Output: Text data (sent to the terminal)

[0682] Step 5:

[0683] The user checks and edits the text data on the terminal. The user opens the text data using a specific text editor (e.g., Notepad++, Visual Studio Code), makes necessary corrections (e.g., correcting typos or replacing terms), and clicks the save button when editing is complete.

[0684] Input: Text data

[0685] Output: Edited text data

[0686] Step 6:

[0687] The device collects the edited text data based on the user's operations, generates JSON format data, and sends it to the server using the HTTPS protocol.

[0688] Input: Edited text data

[0689] Output: None (transmission process)

[0690] Step 7:

[0691] The server saves the edited text data in a database. The server analyzes the received text data and stores it in the database. After saving is complete, the server notifies the terminal of the status.

[0692] Input: Edited text data

[0693] Output: Save successful status

[0694] Step 8:

[0695] To convert all or part of the edited text data into audio data, the user selects the relevant part on the device and clicks the "Convert to audio" button. The device then sends the selected text data to the server.

[0696] Input: Selected text data

[0697] Output: None (transmission process)

[0698] Step 9:

[0699] The server converts the received text data into voice data using a generative AI model. The server then passes the text data to a speech synthesis API (e.g., Amazon Polly) to generate natural-sounding voice data. The generated voice data is then returned to the device in JSON format.

[0700] Input: Text data

[0701] Output: Audio data

[0702] Step 10:

[0703] The device plays the generated audio data. The device analyzes the audio data received from the server and plays it using the built-in audio player. The user can press the play button to listen to the generated audio.

[0704] Input: Audio data

[0705] Output: None (playback processing)

[0706] (Application example 1)

[0707] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0708] Conventional factory robots lack the ability to accurately convert voice instructions into text in real time and provide voice feedback based on the results of text analysis. This makes it difficult for them to communicate smoothly with operators and efficiently execute tasks. The purpose of this invention is to solve this problem and improve productivity and communication within factories.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0710] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and means for analyzing the text data and executing a task, which enables the factory robot to convert voice instructions into text in real time and quickly provide the analysis results as voice feedback to the operator.

[0711] "Voice data" refers to the recording, transmission and processing of human voice as electronic signals.

[0712] "Text data" refers to character string information converted from voice data, or digital character information generated based on that.

[0713] A "generative model" is an algorithm that uses machine learning and artificial intelligence techniques to convert voice data into text data and vice versa.

[0714] "Real-time" is a time concept that means that instructions and data are processed and reflected almost immediately.

[0715] "Analysis" refers to the data processing method used to understand and process the generated text data and take appropriate action or feedback.

[0716] "Feedback" means that the system returns analysis results to the operator in the form of audio data or the like.

[0717] "Speech synthesis" is a technology that converts text data into natural-sounding speech.

[0718] A "server" is a computer system that centrally receives, processes, converts, stores, and transmits audio data.

[0719] An "operator" is a human operator who gives voice instructions to and receives feedback from a factory robot.

[0720] MODE FOR CARRYING OUT THE INVENTION

[0721] Overall system configuration

[0722] This invention is applied to a robot system in a factory, and is a system that converts voice instructions into text data in real time and enables the analysis results to be fed back as voice data. This system mainly consists of the following elements:

[0723] 1. Means of receiving audio data

[0724] This uses the factory robot's microphone to receive voice instructions from the operator.

[0725] 2. Using generative models to convert audio data into text

[0726] Use a speech recognition library (e.g. speech_recognition) to convert the received audio into text.

[0727] 3. A means of analyzing text data and performing tasks

[0728] The converted text data is analyzed and the robot performs the specified task.

[0729] 4. A means of outputting the results of text data analysis as an audio report

[0730] Using a speech synthesis library (e.g. gTTS), the analysis results are generated as natural-sounding speech and reported to the operator.

[0731] This allows operators to give instructions to the robot in real time, enabling it to perform tasks quickly and accurately.

[0732] Hardware and software used

[0733] Hardware:

[0734] Microphone: Collects the operator's voice instructions.

[0735] Speaker: Plays audio feedback.

[0736] software:

[0737] Speech Recognition Library (speech_recognition): Collects voice data and converts it into text data.

[0738] Text-to-Speech Library (gTTS): Used to convert text data into speech data and play the generated speech data.

[0739] Data processing and calculation

[0740] Acquisition and transcription of audio data:

[0741] The microphone receives voice instructions given by the user and transmits them to the server, where the speech recognition library converts the received voice data into text data.

[0742] Analyzing text data and performing tasks:

[0743] The server analyzes the converted text data and determines the corresponding actions for the factory robot to carry out the instructions.

[0744] Audio data generation and feedback:

[0745] Feedback information such as task completion and error messages is generated as text data and converted to voice data using a speech synthesis library. The generated voice data is reported to the operator through a speaker.

[0746] Specific examples

[0747] If the operator gives you the following voice prompts:

[0748] "Please stop the conveyor belt on line 1."

[0749] 1. Speech Recognition:

[0750] The microphone receives the voice and converts it into text data such as "Please stop the conveyor belt on line 1" through a speech recognition library.

[0751] 2. Analysis and execution:

[0752] The server analyzes the text data and instructs the robot to stop the conveyor belt.

[0753] 3. Audio feedback:

[0754] As a result of the analysis, the robot reports to the operator "The conveyor belt has stopped" using voice data generated by a speech synthesis library.

[0755] Prompt Sentence Examples

[0756] Example of an input prompt for a generative AI model:

[0757] Please show me how to convert the following voice instructions into text data and convert the analysis results back into voice:

[0758] Voice: "Please stop the conveyor belt on line 1"

[0759] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0760] Step 1:

[0761] Receiving and sending audio data

[0762] The user acts as an operator and issues instructions into the microphone. The microphone receives the voice data, and the terminal transmits the voice data to the server.

[0763] Specific behavior:

[0764] Input: User's voice data

[0765] Data processing: Capture audio with a microphone and convert it into a digital signal

[0766] Output: Audio data sent to the server

[0767] Step 2:

[0768] Converting audio data to text

[0769] The server converts the received voice data into text data using a voice recognition library.

[0770] Specific behavior:

[0771] Input: Audio data

[0772] Data calculation: Analyzes speech using a speech recognition algorithm (speech_recognition) and converts it into text

[0773] Output: Text data

[0774] Step 3:

[0775] Text data analysis and task instructions

[0776] The server analyzes the converted text data and instructs the robot on the appropriate task.

[0777] Specific behavior:

[0778] Input: Text data

[0779] Data Computing: Using text analysis algorithms to interpret instructions and determine the tasks to be performed

[0780] Output: Instructions to the robot to execute the task

[0781] Step 4:

[0782] Executing a task

[0783] The robot performs actual operations based on task instructions received from the server, such as stopping a conveyor belt.

[0784] Specific behavior:

[0785] Input: Task execution instructions

[0786] Data processing: The robot performs physical actions according to instructions

[0787] Output: Task completion (e.g. conveyor belt stopped)

[0788] Step 5:

[0789] Generate feedback on task completion

[0790] The server confirms the completion of the task and generates the details as text data.

[0791] Specific behavior:

[0792] Input: Task execution result

[0793] Data processing: Convert task completion information into text data

[0794] Output: Text data (e.g. "Conveyor belt has stopped")

[0795] Step 6:

[0796] Text-to-speech conversion

[0797] The text data generated by the server is converted into voice data using a voice synthesis library.

[0798] Specific behavior:

[0799] Input: Text data

[0800] Data Computing: Convert text to speech using a text-to-speech algorithm (gTTS)

[0801] Output: Audio data

[0802] Step 7:

[0803] Playback and reporting of audio data

[0804] The server sends the converted voice data to the terminal and reports it to the user through the speaker.

[0805] Specific behavior:

[0806] Input: Audio data

[0807] Data calculation: The device plays the audio data received on the speaker.

[0808] Output: User confirms audio feedback

[0809] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0810] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide users with highly accurate and natural interactions.

[0811] Overall system configuration

[0812] The system mainly consists of the following components:

[0813] 1. Means of receiving audio data

[0814] 2. Generative model for converting audio data into text data

[0815] 3. Means for editing the converted text data

[0816] 4. How to save edited text data

[0817] 5. Generative model for converting text data into speech data

[0818] 6. Means for playing back converted audio data

[0819] 7. Emotion engine for recognizing user emotions

[0820] The process of converting voice data into text data

[0821] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server then sends the text data to an emotion engine to analyze the user's emotions. Text data tagged with emotion-based tags is generated, stored in a database, and sent to the user's device. The user can review the text data on their device and edit it as necessary.

[0822] Specific examples

[0823] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server. The server will then use a generative model to convert the speech into text. During this process, the audio data is transcribed with high accuracy and text data is generated. The emotion engine will then analyze the user's emotions and assign emotion tags to them. The generated text data will then be displayed to the user, allowing them to correct any errors in specific terms or names.

[0824] Processing to edit text data

[0825] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[0826] Specific examples

[0827] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[0828] The process of converting text data into audio data

[0829] If a user wants to convert part or all of the edited text data into voice data, they select the relevant section on their device and click the "Convert to voice" button. The device then sends the selected text data to the server, which then sends the data to the generative model for voice synthesis. The generated voice data is generated by the emotion engine in a way that reflects the user's emotions, and is then sent to the user's device and played back.

[0830] Specific examples

[0831] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[0832] This concludes the detailed description of the embodiment of the present invention. This system allows users to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[0833] The processing flow will be explained below.

[0834] Converting speech to text and performing sentiment analysis

[0835] Step 1:

[0836] The user clicks on the upload button on the platform to upload the audio file.

[0837] Step 2:

[0838] The device reads the audio file selected by the user and prepares it for transmission to the server.

[0839] Step 3:

[0840] The device sends the audio file to the server as an HTTP POST request.

[0841] Step 4:

[0842] The server checks the received audio file and temporarily saves it.

[0843] Step 5:

[0844] The server sends the audio file to the speech recognition generation model, which starts the process of converting the audio data into text data.

[0845] Step 6:

[0846] The generative model analyzes the audio data, generates text data, and returns it to the server.

[0847] Step 7:

[0848] The server sends the generated text data to the emotion engine.

[0849] Step 8:

[0850] The emotion engine analyzes the text data and tags it with the user's emotions.

[0851] Step 9:

[0852] The server stores emotion-tagged text data in a database and sends it to the user's device.

[0853] Step 10:

[0854] The terminal displays the acquired text data to the user.

[0855] Text editing process

[0856] Step 1:

[0857] The user uses a text editor on the platform to perform editing operations on the text data.

[0858] Step 2:

[0859] The user clicks the save button to save the text data that he has edited.

[0860] Step 3:

[0861] The terminal prepares to send the edited text data to the server.

[0862] Step 4:

[0863] The device sends the edited text data to the server as an HTTP POST request.

[0864] Step 5:

[0865] The server checks the received edited text data and saves it in the database.

[0866] Converting text to speech and reflecting emotions

[0867] Step 1:

[0868] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[0869] Step 2:

[0870] The terminal prepares to send the selected text data to the server.

[0871] Step 3:

[0872] The terminal sends the selected text data to the server as an HTTP POST request.

[0873] Step 4:

[0874] The server checks the received text data and sends it to the speech synthesis generation model.

[0875] Step 5:

[0876] The generative model analyzes the text data, generates audio data, and returns it to the server.

[0877] Step 6:

[0878] The server sends the generated voice data to the emotion engine, where it processes the data to reflect the emotion.

[0879] Step 7:

[0880] The emotion engine adds emotion information to the voice data, and the final voice data is sent to the server.

[0881] Step 8:

[0882] The server transmits the voice data that reflects the emotion to the user's terminal.

[0883] Step 9:

[0884] The device prepares to play the received audio data.

[0885] Step 10:

[0886] The terminal plays the audio data to the user.

[0887] Example 2

[0888] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0889] Conventional systems that convert voice data into text data have limitations in the accuracy of voice recognition and the naturalness of synthesized voice, and one problem is that it is particularly difficult to reflect the user's emotions. This makes it difficult for users to communicate naturally, including their emotions. The present invention aims to solve these problems and realize highly accurate conversion of voice data into text and generation of voice data that reflects emotions.

[0890] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0891] In this invention, the server includes means for receiving voice data, means for using a generative model to convert voice data into text data, means for performing emotion recognition on the converted text data, means for saving emotion-tagged text data, means for saving edited text data, means for using a generative model to convert text data into voice data, and means for playing back the converted voice data. This enables a user to convert voice data into text, edit text data, and generate voice data that reflects emotions with high accuracy and in a natural way.

[0892] "Audio data" refers to data in which audio is recorded in digital format.

[0893] "Text data" refers to data obtained by converting voice data into character information.

[0894] A "generative model" is a machine learning model used to convert voice data to text data or vice versa.

[0895] "Emotion recognition" is a technology that analyzes and identifies a user's emotions from text data and voice data.

[0896] An "emotion tag" is a tag that includes information about an emotion that is assigned to text data by emotion recognition.

[0897] A "server" is a computer system that receives voice data, converts it, recognizes emotions, stores it, and transmits the data.

[0898] A "terminal" is a device that is operated by a user and is used to send and receive voice data and text data.

[0899] "Real time" refers to a processing format in which the process from receiving voice data to generating text data is carried out without delay.

[0900] "Speech synthesis" is a technology that generates synthetic speech based on text data.

[0901] The present invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interaction. The specific operation of this system is as follows.

[0902] Hardware and software used

[0903] Hardware

[0904] User's input device (terminal): PC, smartphone, tablet, etc.

[0905] Server: High-performance cloud or on-premise server

[0906] software

[0907] Speech recognition API: The general name is "Speech recognition API"

[0908] Emotion recognition engine: The generic name "emotion recognition engine"

[0909] Speech synthesis API: The general name is "Speech synthesis API"

[0910] Database: The general term "database software"

[0911] Text editor: A common text editing software installed on the device.

[0912] Overall system configuration

[0913] The system mainly consists of the following components:

[0914] 1. Means of receiving audio data

[0915] 2. Generative model for converting audio data into text data

[0916] 3. Means for editing the converted text data

[0917] 4. Means of Emotion Recognition

[0918] 5. A means of storing emotion-tagged text data

[0919] 6. How to save edited text data

[0920] 7. Generative model for converting text data into speech data

[0921] 8. Means for playing back converted audio data

[0922] Example of operation

[0923] The process of converting voice data into text data

[0924] When a user uploads an audio file, the device sends the file to the server. The server converts the audio data into text using a speech recognition API, and then sends the text to an emotion recognition engine for emotional analysis. Emotion-based tagged text data is generated, stored in a database, and sent to the user's device. The user can then review the text data on their device and edit it as necessary.

[0925] Specific prompt examples:

[0926] User: Upload a 1-hour meeting recording

[0927] Terminal: Send a file to the server

[0928] Server: Convert speech to text using speech recognition API

[0929] Server: Analyze emotions using an emotion recognition engine

[0930] Server: Generate emotion-tagged text data and store it in a database

[0931] Terminal: displays text data to the user

[0932] Processing to edit text data

[0933] The user can edit the text data using a text editor on the terminal. After editing, the text data is sent to the server at the user's instruction and saved in the database again.

[0934] Specific prompt examples:

[0935] User: Open text data in a text editor

[0936] User: Correct typos and omissions

[0937] User: Clicks the Save button to send the changes to the server

[0938] Server: Save the modified text data to the database

[0939] The process of converting text data into audio data

[0940] If a user wants to convert the edited text data into audio data, they select the relevant section on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then uses a speech synthesis API to generate audio data. This audio data is generated using an emotion recognition engine to reflect the user's emotions, and is then sent to the user's device and played back.

[0941] Specific prompt examples:

[0942] User: Select part of edited text data

[0943] User: Clicks the "Convert to Audio" button

[0944] Server: Sends the selected text data to the speech synthesis API

[0945] Server: Generates natural voice using emotion recognition engine

[0946] Server: Sends the generated audio data to the device

[0947] User: Press the play button to check

[0948] The above is a specific embodiment of the program processing of this system. This system allows the user to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[0949] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0950] Step 1: Receiving audio data

[0951] The user selects an audio file on the terminal and clicks the "Upload" button. The input is the audio file selected by the user. The terminal sends this audio file to the server. The output is the audio file sent to the server.

[0952] Step 2: Save the audio data to the server

[0953] The server saves the received audio file in a specific folder. The received audio file is the input, and the saved audio file is the output. At this time, the terminal displays the file transmission status on the screen.

[0954] Step 3: Converting audio data to text data

[0955] The server sends the saved voice data to a voice recognition API (commonly known as a "voice recognition API"). The voice data is input, and the API analyzes it to generate text data. The text data is obtained as output. Specifically, the API returns the text data "We discussed this project."

[0956] Step 4: Emotion Recognition

[0957] The server sends the generated text data to an emotion recognition engine (commonly known as an "emotion recognition engine"). The engine receives text data as input and identifies the user's emotion from the text data. The output is text data tagged with an emotion. Specifically, the emotion engine assigns a tag of "joy."

[0958] Step 5: Saving emotion-tagged text data

[0959] The server stores emotion-tagged text data in a database.,The input is emotion-tagged text data, and the output is the,text data stored in the database.

[0960] Step 6: Sending text data to the user

[0961] The server sends emotion-tagged text data stored in the database to the user's device. The input is text data from the database, and the output is text data sent to the user's device. The user can view this data.

[0962] Step 7: Editing text data

[0963] The user opens the received text data using a text editor on the terminal and makes the necessary corrections. The input is the transmitted text data, and the user performs editing tasks such as correcting typos and omissions. The edited text data is obtained as output. In concrete terms, the user changes "project" to "plan."

[0964] Step 8: Save the edited text data to the server

[0965] The user completes editing the text and clicks the "Save" button. The terminal sends the edited text data to the server. The input is the edited text data, and the server saves it in the database. The output is the corrected text data saved in the database.

[0966] Step 9: Converting text data to audio data

[0967] The user selects the edited text data and clicks the "Convert to speech" button. The device sends the selected text data to the server. The selected text data is input, and the server sends it to a speech synthesis API (commonly known as a "speech synthesis API") to generate speech data. The generated speech data is obtained as output.

[0968] Step 10: Send and play the generated audio data to the user

[0969] The server sends the generated voice data to the user's device. The generated voice data is input, and the voice data sent to the device is obtained as output. The user presses the play button to check the voice. Specifically, the user clicks the play button, and the text is played back in a natural voice.

[0970] (Application example 2)

[0971] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0972] Conventional food delivery services have problems that can easily lead to dissatisfaction due to misreading of order details and lack of emotional response. In particular, when ordering by voice, when users provide emotional feedback, the emotional information is not properly conveyed, making it difficult to improve service quality. Therefore, a system is needed that can accurately convert users' voice input into text and appropriately reflect emotional information, thereby deepening mutual understanding between service providers and users and improving the quality of customer support.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0974] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and an emotion engine for recognizing a user's emotion and reflecting the emotion information in the text data and voice data. This allows the user's voice input to be converted into text data with high accuracy, and makes it possible to appropriately notify the service provider of the user's emotion information.

[0975] 1. "Voice Data" means a digitized recording of a user's speech.

[0976] 2. "Text data" means digital information that has been converted from audio data into textual information.

[0977] 3. A "generative model" is a model that has been trained using machine learning or AI techniques to perform a specific task.

[0978] 4. "Means" are the methods of equipment or software used to achieve a particular purpose.

[0979] 5. "Emotion engine" is a technology that analyzes a user's voice and text data to recognize and tag emotions.

[0980] 6. "Customer Support Team" refers to the department within the service provider that handles user inquiries and resolves problems.

[0981] 7. "Notice" means a communication that conveys information to a specific recipient.

[0982] 8. "Service Provider" means a company or organization that provides food delivery or other services to Users.

[0983] 9. "Real time" means that input and output occur immediately without delay.

[0984] 10. "Speech synthesis function" refers to technology that converts text data into voice data and plays it back as natural-sounding voice.

[0985] MODE FOR CARRYING OUT THE INVENTION

[0986] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. The system integrates voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interactions.

[0987] Overall system configuration

[0988] The system mainly consists of the following components:

[0989] 1. Means of receiving audio data

[0990] 2. Using generative models to convert audio data into text

[0991] 3. Means for editing the converted text data

[0992] 4. How to save edited text data

[0993] 5. Using generative models to convert text data into audio data

[0994] 6. Means for playing back converted audio data

[0995] 7. An emotion engine that recognizes user emotions and reflects that emotional information in text and voice data.

[0996] 8. Means for recognizing emotions based on user voice input and sending notifications containing emotion information to service providers

[0997] The process of converting voice data into text data

[0998] When a user speaks, the device sends this voice data to the server. The server then sends the voice data to a generative model for speech recognition. The generated text data is analyzed by an emotion engine to analyze the user's emotional information, and an emotion tag is added to the text data. This text data is then stored in a database and sent to the user's device.

[0999] Specific examples

[1000] For example, if a user reports by voice that "my order is late and I'm frustrated," this voice data is sent by the device to the server. The server uses a generative model to convert the speech into text, and in the process, the emotion engine recognizes the emotion "frustrated." Then, emotion-tagged text data is generated, stored in a database, and sent to the user's device.

[1001] Processing to edit text data

[1002] When a user edits text data, the user can make corrections using a text editor on the device. Once edited, the text data is sent to the server at the user's instruction and saved in the database again.

[1003] Specific examples

[1004] The user can check the generated text data and correct any typos, omissions, or translation errors. For example, it is possible to change "The order will not arrive quickly" to "The order is delayed." Once the corrections are complete, the user clicks the save button and sends the corrections to the server.

[1005] The process of converting text data into audio data

[1006] If a user wants to convert part or all of the edited text data into speech data, they select the relevant part on their device and click the "Convert to speech" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis.

[1007] Specific examples

[1008] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[1009] Hardware and software used

[1010] The hardware used is a cloud server (e.g., Amazon EC2) for high-speed processing and managing large amounts of data, and a smartphone for user interaction. The software used is a generative model for speech recognition and emotion recognition using PyTorch and Transformers.

[1011] Prompt Sentence Examples

[1012] If a user says, "I'd like to order a pizza and salad," create a program that takes the voice data, converts it to text, recognizes the sentiment, and notifies your customer support team.

[1013] As described above, this system can convert voice data into text with high accuracy, add emotional information, and output it as voice data, thereby improving the quality of customer support for food delivery services.

[1014] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1015] Step 1:

[1016] The user inputs voice orders or inquiries into the terminal. The input voice data is digitized by the terminal and sent to the server, where it is captured into the system.

[1017] Input: User's voice data

[1018] Output: The digitized audio data is sent to the server.

[1019] Step 2:

[1020] The server sends the received speech data to a generative model (speech recognition model) to convert the speech into text data. High-accuracy speech recognition is achieved using models such as PyTorch and Wav2Vec2. The generated text data is then sent to the next processing step.

[1021] Input: Digitized audio data

[1022] Output: Converted text data

[1023] Step 3:

[1024] The server sends the converted text data to the emotion engine for emotion analysis. The emotion engine analyzes the text data, recognizes the user's emotional information, and tags it. The text data with the added emotional information is stored in a database and sent to the user's device.

[1025] Input: Converted text data

[1026] Output: Text data with emotional information added

[1027] Step 4:

[1028] The user can use a text editor on the device to check the generated text data and edit it as necessary. After editing, the text data is sent from the device to the server and saved in the database.

[1029] Input: Text data with emotional information added

[1030] Output: Edited text data

[1031] Step 5:

[1032] When a user wants to convert edited text data into voice data, they select the relevant part on their device and issue a command to convert it into voice. The device then sends the selected text data to the server, which then generates voice data using a generative model (voice synthesis model).

[1033] Input: Edited text data

[1034] Output: Generated audio data

[1035] Step 6:

[1036] The generated voice data is generated in a way that reflects the user's emotional information through the emotion engine. The voice data is sent to the device, and the user can listen to the voice data by pressing the play button.

[1037] Input: Generated audio data

[1038] Output: Audio data sent to the user's device

[1039] Step 7:

[1040] Based on the user's voice input and edits, a means for sending notifications containing emotional information to the service provider is activated, for example, a notification such as "The user is unhappy with the order" is sent to the customer support team in real time.

[1041] Input: Voice input or edited data

[1042] Output: A notification containing emotion information is sent to the service provider.

[1043] The above are the specific processing steps of the system of the present invention. This configuration enables the user's voice input to be converted into text with high accuracy, emotional information to be appropriately reflected, and the quality of customer support to be improved.

[1044] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1045] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1046] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1047] [Third embodiment]

[1048] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1049] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1050] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1051] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1052] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1053] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1054] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1055] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1056] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1057] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1058] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1059] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1060] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[1061] Overall system configuration

[1062] The system mainly consists of the following components:

[1063] 1. Means of receiving audio data

[1064] 2. Generative model for converting audio data into text data

[1065] 3. Means for editing the converted text data

[1066] 4. How to save edited text data

[1067] 5. Generative model for converting text data into speech data

[1068] 6. Means for playing back converted audio data

[1069] The process of converting voice data into text data

[1070] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server stores the converted text data in a database and sends it to the user's device. The user can check the text data on their device and edit it as necessary.

[1071] Specific examples

[1072] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server, which will then use a generative model to convert the speech to text. During this process, the audio data is transcribed with high accuracy and text data is generated. The generated text data is then displayed to the user, allowing them to correct any errors in specific terms or names.

[1073] Processing to edit text data

[1074] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[1075] Specific examples

[1076] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[1077] The process of converting text data into audio data

[1078] If a user wants to convert part or all of the edited text data into audio data, they select the relevant part on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis. The generated audio data is then sent to the user's device and played back.

[1079] Specific examples

[1080] If a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Audio" button. The server sends this text data to a generative model, which generates natural-sounding audio. This audio data is then sent to the device, where the user can press the play button to listen.

[1081] This completes the detailed description of the embodiment of the present invention. This system allows a user to convert voice data into text with high accuracy, edit the text, and output it again as voice data.

[1082] The processing flow will be explained below.

[1083] The process of converting speech to text

[1084] Step 1:

[1085] The user clicks on the upload button on the platform to upload the audio file.

[1086] Step 2:

[1087] The terminal reads the audio file selected by the user and prepares to send the file to the server.

[1088] Step 3:

[1089] The device sends the audio file to the server as an HTTP POST request.

[1090] Step 4:

[1091] The server checks the received audio file and temporarily saves it.

[1092] Step 5:

[1093] The server sends the audio file to the speech recognition generation model and begins the process of converting it into text data.

[1094] Step 6:

[1095] The generative model analyzes the audio data, generates text data, and returns it to the server.

[1096] Step 7:

[1097] The server stores the generated text data in a database and transmits it to the user's terminal.

[1098] Step 8:

[1099] The terminal displays the acquired text data to the user.

[1100] Text editing process

[1101] Step 1:

[1102] The user uses a text editor on the platform to perform editing operations on the text data.

[1103] Step 2:

[1104] The user clicks the save button to save the text data that he has edited.

[1105] Step 3:

[1106] The terminal prepares to send the edited text data to the server.

[1107] Step 4:

[1108] The device sends the edited text data to the server as an HTTP POST request.

[1109] Step 5:

[1110] The server checks the received edited text data and saves it in the database.

[1111] Text-to-speech processing

[1112] Step 1:

[1113] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[1114] Step 2:

[1115] The terminal prepares to send the selected text data to the server.

[1116] Step 3:

[1117] The terminal sends the selected text data to the server as an HTTP POST request.

[1118] Step 4:

[1119] The server checks the received text data and sends it to the speech synthesis generation model.

[1120] Step 5:

[1121] The generative model analyzes the text data, generates audio data, and returns it to the server.

[1122] Step 6:

[1123] The server transmits the generated voice data to the user's terminal.

[1124] Step 7:

[1125] The terminal prepares to play the received audio data.

[1126] Step 8:

[1127] The terminal plays the audio data to the user.

[1128] Example 1

[1129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1130] There is a need for a system that can convert speech data into text data and then convert that text data back into speech data with high accuracy and naturalness, while also being easy for users to operate. However, conventional technologies have not been able to perform this process efficiently or accurately, and have not yet achieved user satisfaction. Furthermore, achieving real-time conversion and natural speech synthesis has also been a challenge.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1132] In this invention, the server includes means for converting voice data into text data using a generative AI model, means for saving the edited text data, and means for converting the edited text data into voice data using a generative AI model, thereby enabling the generation of highly accurate text data from voice data, editing of the text data, and reconversion into natural voice data.

[1133] "Audio File" means a digital file containing audio data that a User uploads to the System.

[1134] A "terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[1135] A "server" is a central computer system that processes voice data, stores text data, and synthesizes voice.

[1136] A "generative AI model" is a machine learning model that uses speech recognition technology to convert voice data into text data and speech synthesis technology to convert text data into voice data.

[1137] "Text data" refers to character string data converted from voice data by a generative AI model.

[1138] "Editing" refers to an operation in which a user corrects or adds to text data using a text editor on a terminal.

[1139] "Saving" refers to the process of storing edited text data in a storage device such as a database.

[1140] "Voice data" means audio data converted from text data using a generative AI model.

[1141] "Playback" refers to the operation of outputting the generated audio data as audio again on the terminal.

[1142] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[1143] Overall system configuration

[1144] The system mainly consists of the following elements:

[1145] 1. A way for users to upload audio files

[1146] 2. How the device sends the audio file to the server

[1147] 3. A means for the server to convert voice data into text data using a generative AI model (e.g., speech recognition API or speech synthesis API).

[1148] 4. A means for users to view and edit text data

[1149] 5. Means by which the device sends edited text data to the server

[1150] 6. How the server stores the edited text data

[1151] 7. A means for the server to convert edited text data into voice data using a generative AI model

[1152] 8. Means for the device to play back the generated audio data

[1153] Hardware and software used

[1154] Terminal: A device such as a computer, smartphone, or tablet that is operated by a user.

[1155] Server: A central processing unit for high-performance computing

[1156] Generative AI models: speech recognition API, speech synthesis API (e.g. Google Cloud Speech-to-Text, Amazon Polly)

[1157] Text editor: Software that allows users to edit text data (e.g., Notepad++, Visual Studio Code)

[1158] Database: A database system for storing text data (e.g., MySQL, PostgreSQL)

[1159] Specific examples of processing

[1160] For example, if a user uploads a one-hour meeting recording to the platform, the process goes like this:

[1161] 1. The user uploads the "Meeting Recording.wav" file from their device. Click the upload button on the screen and select the desired audio file from the file selection dialog.

[1162] 2. The device then sends the received audio file to the server using a secure communication protocol (e.g., HTTPS).

[1163] 3. The server passes the voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to generate highly accurate text data.

[1164] 4. The generated text data is stored on the server, and the converted results are sent to the user's device.

[1165] 5. The user checks the text data on the device and corrects it as necessary using a text editor (e.g., Notepad++ or Visual Studio Code). Correct any errors in specific terms or names.

[1166] 6. Once edited, the text data is sent back to the server from the device.

[1167] 7. The server saves the edited text data and passes it to a speech synthesis API (e.g., Amazon Polly) to convert it into natural-sounding voice data.

[1168] 8. The generated audio data is sent to the terminal, and the user can listen to it by pressing the play button.

[1169] Examples of prompt statements

[1170] Prompt to convert voice data to text data:

[1171] "Please convert the following audio data to text data.\nAudio file path: / path / to / meeting_audio.wav"

[1172] Prompt to save edited text data:

[1173] "Please save the following text data.\nCorrected text data:..."

[1174] Prompt to convert text data to speech data:

[1175] "Please convert the following text data into audio data.\nText data:..."

[1176] This allows text data to be generated from voice data with high accuracy, the text data to be edited by the user, and the text data to be output again as voice data.

[1177] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1178] Step 1:

[1179] The user uploads an audio file. The user selects an audio file from the file selection dialog on the device and clicks the upload button. This causes the device to obtain the path of the selected audio file and confirm that the file is ready.

[1180] Input: Audio file

[1181] Output: Path to the audio file

[1182] Step 2:

[1183] The device sends the audio file to the server. The device reads the audio file based on the file path and sends an HTTP request to the server using the HTTPS protocol. The request includes the binary data of the audio file.

[1184] Input: Path to the audio file

[1185] Output: None (transmission process)

[1186] Step 3:

[1187] The server converts the received audio file into text data using a generative AI model. The server then passes the audio data to a speech recognition API (e.g., Google Cloud Speech-to-Text) for highly accurate transcription. The generated text data is stored in a database.

[1188] Input: Audio file

[1189] Output: Text data

[1190] Step 4:

[1191] The server sends the converted text data to the device, which then generates an HTTP response and returns the transcribed text data in JSON format to the device.

[1192] Input: Text data

[1193] Output: Text data (sent to the terminal)

[1194] Step 5:

[1195] The user checks and edits the text data on the terminal. The user opens the text data using a specific text editor (e.g., Notepad++, Visual Studio Code), makes necessary corrections (e.g., correcting typos or replacing terms), and clicks the save button when editing is complete.

[1196] Input: Text data

[1197] Output: Edited text data

[1198] Step 6:

[1199] The device collects the edited text data based on the user's operations, generates JSON format data, and sends it to the server using the HTTPS protocol.

[1200] Input: Edited text data

[1201] Output: None (transmission process)

[1202] Step 7:

[1203] The server saves the edited text data in a database. The server analyzes the received text data and stores it in the database. After saving is complete, the server notifies the terminal of the status.

[1204] Input: Edited text data

[1205] Output: Save successful status

[1206] Step 8:

[1207] To convert all or part of the edited text data into audio data, the user selects the relevant part on the device and clicks the "Convert to audio" button. The device then sends the selected text data to the server.

[1208] Input: Selected text data

[1209] Output: None (transmission process)

[1210] Step 9:

[1211] The server converts the received text data into voice data using a generative AI model. The server then passes the text data to a speech synthesis API (e.g., Amazon Polly) to generate natural-sounding voice data. The generated voice data is then returned to the device in JSON format.

[1212] Input: Text data

[1213] Output: Audio data

[1214] Step 10:

[1215] The device plays the generated audio data. The device analyzes the audio data received from the server and plays it using the built-in audio player. The user can press the play button to listen to the generated audio.

[1216] Input: Audio data

[1217] Output: None (playback processing)

[1218] (Application example 1)

[1219] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1220] Conventional factory robots lack the ability to accurately convert voice instructions into text in real time and provide voice feedback based on the results of text analysis. This makes it difficult for them to communicate smoothly with operators and efficiently execute tasks. The purpose of this invention is to solve this problem and improve productivity and communication within factories.

[1221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1222] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and means for analyzing the text data and executing a task, which enables the factory robot to convert voice instructions into text in real time and quickly provide the analysis results as voice feedback to the operator.

[1223] "Voice data" refers to the recording, transmission and processing of human voice as electronic signals.

[1224] "Text data" refers to character string information converted from voice data, or digital character information generated based on that.

[1225] A "generative model" is an algorithm that uses machine learning and artificial intelligence techniques to convert voice data into text data and vice versa.

[1226] "Real-time" is a time concept that means that instructions and data are processed and reflected almost immediately.

[1227] "Analysis" refers to the data processing method used to understand and process the generated text data and take appropriate action or feedback.

[1228] "Feedback" means that the system returns analysis results to the operator in the form of audio data or the like.

[1229] "Speech synthesis" is a technology that converts text data into natural-sounding speech.

[1230] A "server" is a computer system that centrally receives, processes, converts, stores, and transmits audio data.

[1231] An "operator" is a human operator who gives voice instructions to and receives feedback from a factory robot.

[1232] MODE FOR CARRYING OUT THE INVENTION

[1233] Overall system configuration

[1234] This invention is applied to a robot system in a factory, and is a system that converts voice instructions into text data in real time and enables the analysis results to be fed back as voice data. This system mainly consists of the following elements:

[1235] 1. Means of receiving audio data

[1236] This uses the factory robot's microphone to receive voice instructions from the operator.

[1237] 2. Using generative models to convert audio data into text

[1238] Use a speech recognition library (e.g. speech_recognition) to convert the received audio into text.

[1239] 3. A means of analyzing text data and performing tasks

[1240] The converted text data is analyzed and the robot performs the specified task.

[1241] 4. A means of outputting the results of text data analysis as an audio report

[1242] Using a speech synthesis library (e.g. gTTS), the analysis results are generated as natural-sounding speech and reported to the operator.

[1243] This allows operators to give instructions to the robot in real time, enabling it to perform tasks quickly and accurately.

[1244] Hardware and software used

[1245] Hardware:

[1246] Microphone: Collects the operator's voice instructions.

[1247] Speaker: Plays audio feedback.

[1248] software:

[1249] Speech Recognition Library (speech_recognition): Collects voice data and converts it into text data.

[1250] Text-to-Speech Library (gTTS): Used to convert text data into speech data and play the generated speech data.

[1251] Data processing and calculation

[1252] Acquisition and transcription of audio data:

[1253] The microphone receives voice instructions given by the user and transmits them to the server, where the speech recognition library converts the received voice data into text data.

[1254] Analyzing text data and performing tasks:

[1255] The server analyzes the converted text data and determines the corresponding actions for the factory robot to carry out the instructions.

[1256] Audio data generation and feedback:

[1257] Feedback information such as task completion and error messages is generated as text data and converted to voice data using a speech synthesis library. The generated voice data is reported to the operator through a speaker.

[1258] Specific examples

[1259] If the operator gives you the following voice prompts:

[1260] "Please stop the conveyor belt on line 1."

[1261] 1. Speech Recognition:

[1262] The microphone receives the voice and converts it into text data such as "Please stop the conveyor belt on line 1" through a speech recognition library.

[1263] 2. Analysis and execution:

[1264] The server analyzes the text data and instructs the robot to stop the conveyor belt.

[1265] 3. Audio feedback:

[1266] As a result of the analysis, the robot reports to the operator "The conveyor belt has stopped" using voice data generated by a speech synthesis library.

[1267] Prompt Sentence Examples

[1268] Example of an input prompt for a generative AI model:

[1269] Please show me how to convert the following voice instructions into text data and convert the analysis results back into voice:

[1270] Voice: "Please stop the conveyor belt on line 1"

[1271] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1272] Step 1:

[1273] Receiving and sending audio data

[1274] The user acts as an operator and issues instructions into the microphone. The microphone receives the voice data, and the terminal transmits the voice data to the server.

[1275] Specific behavior:

[1276] Input: User's voice data

[1277] Data processing: Capture audio with a microphone and convert it into a digital signal

[1278] Output: Audio data sent to the server

[1279] Step 2:

[1280] Converting audio data to text

[1281] The server converts the received voice data into text data using a voice recognition library.

[1282] Specific behavior:

[1283] Input: Audio data

[1284] Data calculation: Analyzes speech using a speech recognition algorithm (speech_recognition) and converts it into text

[1285] Output: Text data

[1286] Step 3:

[1287] Text data analysis and task instructions

[1288] The server analyzes the converted text data and instructs the robot on the appropriate task.

[1289] Specific behavior:

[1290] Input: Text data

[1291] Data Computing: Using text analysis algorithms to interpret instructions and determine the tasks to be performed

[1292] Output: Instructions to the robot to execute the task

[1293] Step 4:

[1294] Executing a task

[1295] The robot performs actual operations based on task instructions received from the server, such as stopping a conveyor belt.

[1296] Specific behavior:

[1297] Input: Task execution instructions

[1298] Data processing: The robot performs physical actions according to instructions

[1299] Output: Task completion (e.g. conveyor belt stopped)

[1300] Step 5:

[1301] Generate feedback on task completion

[1302] The server confirms the completion of the task and generates the details as text data.

[1303] Specific behavior:

[1304] Input: Task execution result

[1305] Data processing: Convert task completion information into text data

[1306] Output: Text data (e.g. "Conveyor belt has stopped")

[1307] Step 6:

[1308] Text-to-speech conversion

[1309] The text data generated by the server is converted into voice data using a voice synthesis library.

[1310] Specific behavior:

[1311] Input: Text data

[1312] Data Computing: Convert text to speech using a text-to-speech algorithm (gTTS)

[1313] Output: Audio data

[1314] Step 7:

[1315] Playback and reporting of audio data

[1316] The server sends the converted voice data to the terminal and reports it to the user through the speaker.

[1317] Specific behavior:

[1318] Input: Audio data

[1319] Data calculation: The device plays the audio data received on the speaker.

[1320] Output: User confirms audio feedback

[1321] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1322] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide users with highly accurate and natural interactions.

[1323] Overall system configuration

[1324] The system mainly consists of the following components:

[1325] 1. Means of receiving audio data

[1326] 2. Generative model for converting audio data into text data

[1327] 3. Means for editing the converted text data

[1328] 4. How to save edited text data

[1329] 5. Generative model for converting text data into speech data

[1330] 6. Means for playing back converted audio data

[1331] 7. Emotion engine for recognizing user emotions

[1332] The process of converting voice data into text data

[1333] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server then sends the text data to an emotion engine to analyze the user's emotions. Text data tagged with emotion-based tags is generated, stored in a database, and sent to the user's device. The user can review the text data on their device and edit it as necessary.

[1334] Specific examples

[1335] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server. The server will then use a generative model to convert the speech into text. During this process, the audio data is transcribed with high accuracy and text data is generated. The emotion engine will then analyze the user's emotions and assign emotion tags to them. The generated text data will then be displayed to the user, allowing them to correct any errors in specific terms or names.

[1336] Processing to edit text data

[1337] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[1338] Specific examples

[1339] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[1340] The process of converting text data into audio data

[1341] If a user wants to convert part or all of the edited text data into voice data, they select the relevant section on their device and click the "Convert to voice" button. The device then sends the selected text data to the server, which then sends the data to the generative model for voice synthesis. The generated voice data is generated by the emotion engine in a way that reflects the user's emotions, and is then sent to the user's device and played back.

[1342] Specific examples

[1343] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[1344] This concludes the detailed description of the embodiment of the present invention. This system allows users to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[1345] The processing flow will be explained below.

[1346] Converting speech to text and performing sentiment analysis

[1347] Step 1:

[1348] The user clicks on the upload button on the platform to upload the audio file.

[1349] Step 2:

[1350] The device reads the audio file selected by the user and prepares it for transmission to the server.

[1351] Step 3:

[1352] The device sends the audio file to the server as an HTTP POST request.

[1353] Step 4:

[1354] The server checks the received audio file and temporarily saves it.

[1355] Step 5:

[1356] The server sends the audio file to the speech recognition generation model, which starts the process of converting the audio data into text data.

[1357] Step 6:

[1358] The generative model analyzes the audio data, generates text data, and returns it to the server.

[1359] Step 7:

[1360] The server sends the generated text data to the emotion engine.

[1361] Step 8:

[1362] The emotion engine analyzes the text data and tags it with the user's emotions.

[1363] Step 9:

[1364] The server stores emotion-tagged text data in a database and sends it to the user's device.

[1365] Step 10:

[1366] The terminal displays the acquired text data to the user.

[1367] Text editing process

[1368] Step 1:

[1369] The user uses a text editor on the platform to perform editing operations on the text data.

[1370] Step 2:

[1371] The user clicks the save button to save the text data that he has edited.

[1372] Step 3:

[1373] The terminal prepares to send the edited text data to the server.

[1374] Step 4:

[1375] The device sends the edited text data to the server as an HTTP POST request.

[1376] Step 5:

[1377] The server checks the received edited text data and saves it in the database.

[1378] Converting text to speech and reflecting emotions

[1379] Step 1:

[1380] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[1381] Step 2:

[1382] The terminal prepares to send the selected text data to the server.

[1383] Step 3:

[1384] The terminal sends the selected text data to the server as an HTTP POST request.

[1385] Step 4:

[1386] The server checks the received text data and sends it to the speech synthesis generation model.

[1387] Step 5:

[1388] The generative model analyzes the text data, generates audio data, and returns it to the server.

[1389] Step 6:

[1390] The server sends the generated voice data to the emotion engine, where it processes the data to reflect the emotion.

[1391] Step 7:

[1392] The emotion engine adds emotion information to the voice data, and the final voice data is sent to the server.

[1393] Step 8:

[1394] The server transmits the voice data that reflects the emotion to the user's terminal.

[1395] Step 9:

[1396] The device prepares to play the received audio data.

[1397] Step 10:

[1398] The terminal plays the audio data to the user.

[1399] Example 2

[1400] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1401] Conventional systems that convert voice data into text data have limitations in the accuracy of voice recognition and the naturalness of synthesized voice, and one problem is that it is particularly difficult to reflect the user's emotions. This makes it difficult for users to communicate naturally, including their emotions. The present invention aims to solve these problems and realize highly accurate conversion of voice data into text and generation of voice data that reflects emotions.

[1402] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1403] In this invention, the server includes means for receiving voice data, means for using a generative model to convert voice data into text data, means for performing emotion recognition on the converted text data, means for saving emotion-tagged text data, means for saving edited text data, means for using a generative model to convert text data into voice data, and means for playing back the converted voice data. This enables a user to convert voice data into text, edit text data, and generate voice data that reflects emotions with high accuracy and in a natural way.

[1404] "Audio data" refers to data in which audio is recorded in digital format.

[1405] "Text data" refers to data obtained by converting voice data into character information.

[1406] A "generative model" is a machine learning model used to convert voice data to text data or vice versa.

[1407] "Emotion recognition" is a technology that analyzes and identifies a user's emotions from text data and voice data.

[1408] An "emotion tag" is a tag that includes information about an emotion that is assigned to text data by emotion recognition.

[1409] A "server" is a computer system that receives voice data, converts it, recognizes emotions, stores it, and transmits the data.

[1410] A "terminal" is a device that is operated by a user and is used to send and receive voice data and text data.

[1411] "Real time" refers to a processing format in which the process from receiving voice data to generating text data is carried out without delay.

[1412] "Speech synthesis" is a technology that generates synthetic speech based on text data.

[1413] The present invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interaction. The specific operation of this system is as follows.

[1414] Hardware and software used

[1415] Hardware

[1416] User's input device (terminal): PC, smartphone, tablet, etc.

[1417] Server: High-performance cloud or on-premise server

[1418] software

[1419] Speech recognition API: The general name is "Speech recognition API"

[1420] Emotion recognition engine: The generic name "emotion recognition engine"

[1421] Speech synthesis API: The general name is "Speech synthesis API"

[1422] Database: The general term "database software"

[1423] Text editor: A common text editing software installed on the device.

[1424] Overall system configuration

[1425] The system mainly consists of the following components:

[1426] 1. Means of receiving audio data

[1427] 2. Generative model for converting audio data into text data

[1428] 3. Means for editing the converted text data

[1429] 4. Means of Emotion Recognition

[1430] 5. A means of storing emotion-tagged text data

[1431] 6. How to save edited text data

[1432] 7. Generative model for converting text data into speech data

[1433] 8. Means for playing back converted audio data

[1434] Example of operation

[1435] The process of converting voice data into text data

[1436] When a user uploads an audio file, the device sends the file to the server. The server converts the audio data into text using a speech recognition API, and then sends the text to an emotion recognition engine for emotional analysis. Emotion-based tagged text data is generated, stored in a database, and sent to the user's device. The user can then review the text data on their device and edit it as necessary.

[1437] Specific prompt examples:

[1438] User: Upload a 1-hour meeting recording

[1439] Terminal: Send a file to the server

[1440] Server: Convert speech to text using speech recognition API

[1441] Server: Analyze emotions using an emotion recognition engine

[1442] Server: Generate emotion-tagged text data and store it in a database

[1443] Terminal: displays text data to the user

[1444] Processing to edit text data

[1445] The user can edit the text data using a text editor on the terminal. After editing, the text data is sent to the server at the user's instruction and saved in the database again.

[1446] Specific prompt examples:

[1447] User: Open text data in a text editor

[1448] User: Correct typos and omissions

[1449] User: Clicks the Save button to send the changes to the server

[1450] Server: Save the modified text data to the database

[1451] The process of converting text data into audio data

[1452] If a user wants to convert the edited text data into audio data, they select the relevant section on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then uses a speech synthesis API to generate audio data. This audio data is generated using an emotion recognition engine to reflect the user's emotions, and is then sent to the user's device and played back.

[1453] Specific prompt examples:

[1454] User: Select part of edited text data

[1455] User: Clicks the "Convert to Audio" button

[1456] Server: Sends the selected text data to the speech synthesis API

[1457] Server: Generates natural voice using emotion recognition engine

[1458] Server: Sends the generated audio data to the device

[1459] User: Press the play button to check

[1460] The above is a specific embodiment of the program processing of this system. This system allows the user to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[1461] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1462] Step 1: Receiving audio data

[1463] The user selects an audio file on the terminal and clicks the "Upload" button. The input is the audio file selected by the user. The terminal sends this audio file to the server. The output is the audio file sent to the server.

[1464] Step 2: Save the audio data to the server

[1465] The server saves the received audio file in a specific folder. The received audio file is the input, and the saved audio file is the output. At this time, the terminal displays the file transmission status on the screen.

[1466] Step 3: Converting audio data to text data

[1467] The server sends the saved voice data to a voice recognition API (commonly known as a "voice recognition API"). The voice data is input, and the API analyzes it to generate text data. The text data is obtained as output. Specifically, the API returns the text data "We discussed this project."

[1468] Step 4: Emotion Recognition

[1469] The server sends the generated text data to an emotion recognition engine (commonly known as an "emotion recognition engine"). The engine receives text data as input and identifies the user's emotion from the text data. The output is text data tagged with an emotion. Specifically, the emotion engine assigns a tag of "joy."

[1470] Step 5: Saving emotion-tagged text data

[1471] The server stores emotion-tagged text data in a database.,The input is emotion-tagged text data, and the output is the,text data stored in the database.

[1472] Step 6: Sending text data to the user

[1473] The server sends emotion-tagged text data stored in the database to the user's device. The input is text data from the database, and the output is text data sent to the user's device. The user can view this data.

[1474] Step 7: Editing text data

[1475] The user opens the received text data using a text editor on the terminal and makes the necessary corrections. The input is the transmitted text data, and the user performs editing tasks such as correcting typos and omissions. The edited text data is obtained as output. In concrete terms, the user changes "project" to "plan."

[1476] Step 8: Save the edited text data to the server

[1477] The user completes editing the text and clicks the "Save" button. The terminal sends the edited text data to the server. The input is the edited text data, and the server saves it in the database. The output is the corrected text data saved in the database.

[1478] Step 9: Converting text data to audio data

[1479] The user selects the edited text data and clicks the "Convert to speech" button. The device sends the selected text data to the server. The selected text data is input, and the server sends it to a speech synthesis API (commonly known as a "speech synthesis API") to generate speech data. The generated speech data is obtained as output.

[1480] Step 10: Send and play the generated audio data to the user

[1481] The server sends the generated voice data to the user's device. The generated voice data is input, and the voice data sent to the device is obtained as output. The user presses the play button to check the voice. Specifically, the user clicks the play button, and the text is played back in a natural voice.

[1482] (Application example 2)

[1483] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1484] Conventional food delivery services have problems that can easily lead to dissatisfaction due to misreading of order details and lack of emotional response. In particular, when ordering by voice, when users provide emotional feedback, the emotional information is not properly conveyed, making it difficult to improve service quality. Therefore, a system is needed that can accurately convert users' voice input into text and appropriately reflect emotional information, thereby deepening mutual understanding between service providers and users and improving the quality of customer support.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1486] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and an emotion engine for recognizing a user's emotion and reflecting the emotion information in the text data and voice data. This allows the user's voice input to be converted into text data with high accuracy, and makes it possible to appropriately notify the service provider of the user's emotion information.

[1487] 1. "Voice Data" means a digitized recording of a user's speech.

[1488] 2. "Text data" means digital information that has been converted from audio data into textual information.

[1489] 3. A "generative model" is a model that has been trained using machine learning or AI techniques to perform a specific task.

[1490] 4. "Means" are the methods of equipment or software used to achieve a particular purpose.

[1491] 5. "Emotion engine" is a technology that analyzes a user's voice and text data to recognize and tag emotions.

[1492] 6. "Customer Support Team" refers to the department within the service provider that handles user inquiries and resolves problems.

[1493] 7. "Notice" means a communication that conveys information to a specific recipient.

[1494] 8. "Service Provider" means a company or organization that provides food delivery or other services to Users.

[1495] 9. "Real time" means that input and output occur immediately without delay.

[1496] 10. "Speech synthesis function" refers to technology that converts text data into voice data and plays it back as natural-sounding voice.

[1497] MODE FOR CARRYING OUT THE INVENTION

[1498] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. The system integrates voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interactions.

[1499] Overall system configuration

[1500] The system mainly consists of the following components:

[1501] 1. Means of receiving audio data

[1502] 2. Using generative models to convert audio data into text

[1503] 3. Means for editing the converted text data

[1504] 4. How to save edited text data

[1505] 5. Using generative models to convert text data into audio data

[1506] 6. Means for playing back converted audio data

[1507] 7. An emotion engine that recognizes user emotions and reflects that emotional information in text and voice data.

[1508] 8. Means for recognizing emotions based on user voice input and sending notifications containing emotion information to service providers

[1509] The process of converting voice data into text data

[1510] When a user speaks, the device sends this voice data to the server. The server then sends the voice data to a generative model for speech recognition. The generated text data is analyzed by an emotion engine to analyze the user's emotional information, and an emotion tag is added to the text data. This text data is then stored in a database and sent to the user's device.

[1511] Specific examples

[1512] For example, if a user reports by voice that "my order is late and I'm frustrated," this voice data is sent by the device to the server. The server uses a generative model to convert the speech into text, and in the process, the emotion engine recognizes the emotion "frustrated." Then, emotion-tagged text data is generated, stored in a database, and sent to the user's device.

[1513] Processing to edit text data

[1514] When a user edits text data, the user can make corrections using a text editor on the device. Once edited, the text data is sent to the server at the user's instruction and saved in the database again.

[1515] Specific examples

[1516] The user can check the generated text data and correct any typos, omissions, or translation errors. For example, it is possible to change "The order will not arrive quickly" to "The order is delayed." Once the corrections are complete, the user clicks the save button and sends the corrections to the server.

[1517] The process of converting text data into audio data

[1518] If a user wants to convert part or all of the edited text data into speech data, they select the relevant part on their device and click the "Convert to speech" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis.

[1519] Specific examples

[1520] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[1521] Hardware and software used

[1522] The hardware used is a cloud server (e.g., Amazon EC2) for high-speed processing and managing large amounts of data, and a smartphone for user interaction. The software used is a generative model for speech recognition and emotion recognition using PyTorch and Transformers.

[1523] Prompt Sentence Examples

[1524] If a user says, "I'd like to order a pizza and salad," create a program that takes the voice data, converts it to text, recognizes the sentiment, and notifies your customer support team.

[1525] As described above, this system can convert voice data into text with high accuracy, add emotional information, and output it as voice data, thereby improving the quality of customer support for food delivery services.

[1526] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1527] Step 1:

[1528] The user inputs voice orders or inquiries into the terminal. The input voice data is digitized by the terminal and sent to the server, where it is captured into the system.

[1529] Input: User's voice data

[1530] Output: The digitized audio data is sent to the server.

[1531] Step 2:

[1532] The server sends the received speech data to a generative model (speech recognition model) to convert the speech into text data. High-accuracy speech recognition is achieved using models such as PyTorch and Wav2Vec2. The generated text data is then sent to the next processing step.

[1533] Input: Digitized audio data

[1534] Output: Converted text data

[1535] Step 3:

[1536] The server sends the converted text data to the emotion engine for emotion analysis. The emotion engine analyzes the text data, recognizes the user's emotional information, and tags it. The text data with the added emotional information is stored in a database and sent to the user's device.

[1537] Input: Converted text data

[1538] Output: Text data with emotional information added

[1539] Step 4:

[1540] The user can use a text editor on the device to check the generated text data and edit it as necessary. After editing, the text data is sent from the device to the server and saved in the database.

[1541] Input: Text data with emotional information added

[1542] Output: Edited text data

[1543] Step 5:

[1544] When a user wants to convert edited text data into voice data, they select the relevant part on their device and issue a command to convert it into voice. The device then sends the selected text data to the server, which then generates voice data using a generative model (voice synthesis model).

[1545] Input: Edited text data

[1546] Output: Generated audio data

[1547] Step 6:

[1548] The generated voice data is generated in a way that reflects the user's emotional information through the emotion engine. The voice data is sent to the device, and the user can listen to the voice data by pressing the play button.

[1549] Input: Generated audio data

[1550] Output: Audio data sent to the user's device

[1551] Step 7:

[1552] Based on the user's voice input and edits, a means for sending notifications containing emotional information to the service provider is activated, for example, a notification such as "The user is unhappy with the order" is sent to the customer support team in real time.

[1553] Input: Voice input or edited data

[1554] Output: A notification containing emotion information is sent to the service provider.

[1555] The above are the specific processing steps of the system of the present invention. This configuration enables the user's voice input to be converted into text with high accuracy, emotional information to be appropriately reflected, and the quality of customer support to be improved.

[1556] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1557] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1558] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1559] [Fourth embodiment]

[1560] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1561] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1562] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1563] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1564] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1565] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1566] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1567] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1568] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1569] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1570] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1571] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1572] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1573] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[1574] Overall system configuration

[1575] The system mainly consists of the following components:

[1576] 1. Means of receiving audio data

[1577] 2. Generative model for converting audio data into text data

[1578] 3. Means for editing the converted text data

[1579] 4. How to save edited text data

[1580] 5. Generative model for converting text data into speech data

[1581] 6. Means for playing back converted audio data

[1582] The process of converting voice data into text data

[1583] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server stores the converted text data in a database and sends it to the user's device. The user can check the text data on their device and edit it as necessary.

[1584] Specific examples

[1585] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server, which will then use a generative model to convert the speech to text. During this process, the audio data is transcribed with high accuracy and text data is generated. The generated text data is then displayed to the user, allowing them to correct any errors in specific terms or names.

[1586] Processing to edit text data

[1587] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[1588] Specific examples

[1589] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[1590] The process of converting text data into audio data

[1591] If a user wants to convert part or all of the edited text data into audio data, they select the relevant part on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis. The generated audio data is then sent to the user's device and played back.

[1592] Specific examples

[1593] If a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Audio" button. The server sends this text data to a generative model, which generates natural-sounding audio. This audio data is then sent to the device, where the user can press the play button to listen.

[1594] This completes the detailed description of the embodiment of the present invention. This system allows a user to convert voice data into text with high accuracy, edit the text, and output it again as voice data.

[1595] The processing flow will be explained below.

[1596] The process of converting speech to text

[1597] Step 1:

[1598] The user clicks on the upload button on the platform to upload the audio file.

[1599] Step 2:

[1600] The terminal reads the audio file selected by the user and prepares to send the file to the server.

[1601] Step 3:

[1602] The device sends the audio file to the server as an HTTP POST request.

[1603] Step 4:

[1604] The server checks the received audio file and temporarily saves it.

[1605] Step 5:

[1606] The server sends the audio file to the speech recognition generation model and begins the process of converting it into text data.

[1607] Step 6:

[1608] The generative model analyzes the audio data, generates text data, and returns it to the server.

[1609] Step 7:

[1610] The server stores the generated text data in a database and transmits it to the user's terminal.

[1611] Step 8:

[1612] The terminal displays the acquired text data to the user.

[1613] Text editing process

[1614] Step 1:

[1615] The user uses a text editor on the platform to perform editing operations on the text data.

[1616] Step 2:

[1617] The user clicks the save button to save the text data that he has edited.

[1618] Step 3:

[1619] The terminal prepares to send the edited text data to the server.

[1620] Step 4:

[1621] The device sends the edited text data to the server as an HTTP POST request.

[1622] Step 5:

[1623] The server checks the received edited text data and saves it in the database.

[1624] Text-to-speech processing

[1625] Step 1:

[1626] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[1627] Step 2:

[1628] The terminal prepares to send the selected text data to the server.

[1629] Step 3:

[1630] The terminal sends the selected text data to the server as an HTTP POST request.

[1631] Step 4:

[1632] The server checks the received text data and sends it to the speech synthesis generation model.

[1633] Step 5:

[1634] The generative model analyzes the text data, generates audio data, and returns it to the server.

[1635] Step 6:

[1636] The server transmits the generated voice data to the user's terminal.

[1637] Step 7:

[1638] The terminal prepares to play the received audio data.

[1639] Step 8:

[1640] The terminal plays the audio data to the user.

[1641] Example 1

[1642] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1643] There is a need for a system that can convert speech data into text data and then convert that text data back into speech data with high accuracy and naturalness, while also being easy for users to operate. However, conventional technologies have not been able to perform this process efficiently or accurately, and have not yet achieved user satisfaction. Furthermore, achieving real-time conversion and natural speech synthesis has also been a challenge.

[1644] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1645] In this invention, the server includes means for converting voice data into text data using a generative AI model, means for saving the edited text data, and means for converting the edited text data into voice data using a generative AI model, thereby enabling the generation of highly accurate text data from voice data, editing of the text data, and reconversion into natural voice data.

[1646] "Audio File" means a digital file containing audio data that a User uploads to the System.

[1647] A "terminal" is a device operated by a user, such as a computer, smartphone, or tablet.

[1648] A "server" is a central computer system that processes voice data, stores text data, and synthesizes voice.

[1649] A "generative AI model" is a machine learning model that uses speech recognition technology to convert voice data into text data and speech synthesis technology to convert text data into voice data.

[1650] "Text data" refers to character string data converted from voice data by a generative AI model.

[1651] "Editing" refers to an operation in which a user corrects or adds to text data using a text editor on a terminal.

[1652] "Saving" refers to the process of storing edited text data in a storage device such as a database.

[1653] "Voice data" means audio data converted from text data using a generative AI model.

[1654] "Playback" refers to the operation of outputting the generated audio data as audio again on the terminal.

[1655] The present invention is a system that can convert voice data into text data and then convert the text data back into voice data. This system utilizes both voice recognition and voice synthesis technologies to provide users with high accuracy and naturalness.

[1656] Overall system configuration

[1657] The system mainly consists of the following elements:

[1658] 1. A way for users to upload audio files

[1659] 2. How the device sends the audio file to the server

[1660] 3. A means for the server to convert voice data into text data using a generative AI model (e.g., speech recognition API or speech synthesis API).

[1661] 4. A means for users to view and edit text data

[1662] 5. Means by which the device sends edited text data to the server

[1663] 6. How the server stores the edited text data

[1664] 7. A means for the server to convert edited text data into voice data using a generative AI model

[1665] 8. Means for the device to play back the generated audio data

[1666] Hardware and software used

[1667] Terminal: A device such as a computer, smartphone, or tablet that is operated by a user.

[1668] Server: A central processing unit for high-performance computing

[1669] Generative AI models: speech recognition API, speech synthesis API (e.g. Google Cloud Speech-to-Text, Amazon Polly)

[1670] Text editor: Software that allows users to edit text data (e.g., Notepad++, Visual Studio Code)

[1671] Database: A database system for storing text data (e.g., MySQL, PostgreSQL)

[1672] Specific examples of processing

[1673] For example, if a user uploads a one-hour meeting recording to the platform, the process goes like this:

[1674] 1. The user uploads the "Meeting Recording.wav" file from their device. Click the upload button on the screen and select the desired audio file from the file selection dialog.

[1675] 2. The device then sends the received audio file to the server using a secure communication protocol (e.g., HTTPS).

[1676] 3. The server passes the voice data to a speech recognition API (e.g., Google Cloud Speech-to-Text) to generate highly accurate text data.

[1677] 4. The generated text data is stored on the server, and the converted results are sent to the user's device.

[1678] 5. The user checks the text data on the device and corrects it as necessary using a text editor (e.g., Notepad++ or Visual Studio Code). Correct any errors in specific terms or names.

[1679] 6. Once edited, the text data is sent back to the server from the device.

[1680] 7. The server saves the edited text data and passes it to a speech synthesis API (e.g., Amazon Polly) to convert it into natural-sounding voice data.

[1681] 8. The generated audio data is sent to the terminal, and the user can listen to it by pressing the play button.

[1682] Examples of prompt statements

[1683] Prompt to convert voice data to text data:

[1684] "Please convert the following audio data to text data.\nAudio file path: / path / to / meeting_audio.wav"

[1685] Prompt to save edited text data:

[1686] "Please save the following text data.\nCorrected text data:..."

[1687] Prompt to convert text data to speech data:

[1688] "Please convert the following text data into audio data.\nText data:..."

[1689] This allows text data to be generated from voice data with high accuracy, the text data to be edited by the user, and the text data to be output again as voice data.

[1690] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1691] Step 1:

[1692] The user uploads an audio file. The user selects an audio file from the file selection dialog on the device and clicks the upload button. This causes the device to obtain the path of the selected audio file and confirm that the file is ready.

[1693] Input: Audio file

[1694] Output: Path to the audio file

[1695] Step 2:

[1696] The device sends the audio file to the server. The device reads the audio file based on the file path and sends an HTTP request to the server using the HTTPS protocol. The request includes the binary data of the audio file.

[1697] Input: Path to the audio file

[1698] Output: None (transmission process)

[1699] Step 3:

[1700] The server converts the received audio file into text data using a generative AI model. The server then passes the audio data to a speech recognition API (e.g., Google Cloud Speech-to-Text) for highly accurate transcription. The generated text data is stored in a database.

[1701] Input: Audio file

[1702] Output: Text data

[1703] Step 4:

[1704] The server sends the converted text data to the device, which then generates an HTTP response and returns the transcribed text data in JSON format to the device.

[1705] Input: Text data

[1706] Output: Text data (sent to the terminal)

[1707] Step 5:

[1708] The user checks and edits the text data on the terminal. The user opens the text data using a specific text editor (e.g., Notepad++, Visual Studio Code), makes necessary corrections (e.g., correcting typos or replacing terms), and clicks the save button when editing is complete.

[1709] Input: Text data

[1710] Output: Edited text data

[1711] Step 6:

[1712] The device collects the edited text data based on the user's operations, generates JSON format data, and sends it to the server using the HTTPS protocol.

[1713] Input: Edited text data

[1714] Output: None (transmission process)

[1715] Step 7:

[1716] The server saves the edited text data in a database. The server analyzes the received text data and stores it in the database. After saving is complete, the server notifies the terminal of the status.

[1717] Input: Edited text data

[1718] Output: Save successful status

[1719] Step 8:

[1720] To convert all or part of the edited text data into audio data, the user selects the relevant part on the device and clicks the "Convert to audio" button. The device then sends the selected text data to the server.

[1721] Input: Selected text data

[1722] Output: None (transmission process)

[1723] Step 9:

[1724] The server converts the received text data into voice data using a generative AI model. The server then passes the text data to a speech synthesis API (e.g., Amazon Polly) to generate natural-sounding voice data. The generated voice data is then returned to the device in JSON format.

[1725] Input: Text data

[1726] Output: Audio data

[1727] Step 10:

[1728] The device plays the generated audio data. The device analyzes the audio data received from the server and plays it using the built-in audio player. The user can press the play button to listen to the generated audio.

[1729] Input: Audio data

[1730] Output: None (playback processing)

[1731] (Application example 1)

[1732] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1733] Conventional factory robots lack the ability to accurately convert voice instructions into text in real time and provide voice feedback based on the results of text analysis. This makes it difficult for them to communicate smoothly with operators and efficiently execute tasks. The purpose of this invention is to solve this problem and improve productivity and communication within factories.

[1734] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1735] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and means for analyzing the text data and executing a task, which enables the factory robot to convert voice instructions into text in real time and quickly provide the analysis results as voice feedback to the operator.

[1736] "Voice data" refers to the recording, transmission and processing of human voice as electronic signals.

[1737] "Text data" refers to character string information converted from voice data, or digital character information generated based on that.

[1738] A "generative model" is an algorithm that uses machine learning and artificial intelligence techniques to convert voice data into text data and vice versa.

[1739] "Real-time" is a time concept that means that instructions and data are processed and reflected almost immediately.

[1740] "Analysis" refers to the data processing method used to understand and process the generated text data and take appropriate action or feedback.

[1741] "Feedback" means that the system returns analysis results to the operator in the form of audio data or the like.

[1742] "Speech synthesis" is a technology that converts text data into natural-sounding speech.

[1743] A "server" is a computer system that centrally receives, processes, converts, stores, and transmits audio data.

[1744] An "operator" is a human operator who gives voice instructions to and receives feedback from a factory robot.

[1745] MODE FOR CARRYING OUT THE INVENTION

[1746] Overall system configuration

[1747] This invention is applied to a robot system in a factory, and is a system that converts voice instructions into text data in real time and enables the analysis results to be fed back as voice data. This system mainly consists of the following elements:

[1748] 1. Means of receiving audio data

[1749] This uses the factory robot's microphone to receive voice instructions from the operator.

[1750] 2. Using generative models to convert audio data into text

[1751] Use a speech recognition library (e.g. speech_recognition) to convert the received audio into text.

[1752] 3. A means of analyzing text data and performing tasks

[1753] The converted text data is analyzed and the robot performs the specified task.

[1754] 4. A means of outputting the results of text data analysis as an audio report

[1755] Using a speech synthesis library (e.g. gTTS), the analysis results are generated as natural-sounding speech and reported to the operator.

[1756] This allows operators to give instructions to the robot in real time, enabling it to perform tasks quickly and accurately.

[1757] Hardware and software used

[1758] Hardware:

[1759] Microphone: Collects the operator's voice instructions.

[1760] Speaker: Plays audio feedback.

[1761] software:

[1762] Speech Recognition Library (speech_recognition): Collects voice data and converts it into text data.

[1763] Text-to-Speech Library (gTTS): Used to convert text data into speech data and play the generated speech data.

[1764] Data processing and calculation

[1765] Acquisition and transcription of audio data:

[1766] The microphone receives voice instructions given by the user and transmits them to the server, where the speech recognition library converts the received voice data into text data.

[1767] Analyzing text data and performing tasks:

[1768] The server analyzes the converted text data and determines the corresponding actions for the factory robot to carry out the instructions.

[1769] Audio data generation and feedback:

[1770] Feedback information such as task completion and error messages is generated as text data and converted to voice data using a speech synthesis library. The generated voice data is reported to the operator through a speaker.

[1771] Specific examples

[1772] If the operator gives you the following voice prompts:

[1773] "Please stop the conveyor belt on line 1."

[1774] 1. Speech Recognition:

[1775] The microphone receives the voice and converts it into text data such as "Please stop the conveyor belt on line 1" through a speech recognition library.

[1776] 2. Analysis and execution:

[1777] The server analyzes the text data and instructs the robot to stop the conveyor belt.

[1778] 3. Audio feedback:

[1779] As a result of the analysis, the robot reports to the operator "The conveyor belt has stopped" using voice data generated by a speech synthesis library.

[1780] Prompt Sentence Examples

[1781] Example of an input prompt for a generative AI model:

[1782] Please show me how to convert the following voice instructions into text data and convert the analysis results back into voice:

[1783] Voice: "Please stop the conveyor belt on line 1"

[1784] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1785] Step 1:

[1786] Receiving and sending audio data

[1787] The user acts as an operator and issues instructions into the microphone. The microphone receives the voice data, and the terminal transmits the voice data to the server.

[1788] Specific behavior:

[1789] Input: User's voice data

[1790] Data processing: Capture audio with a microphone and convert it into a digital signal

[1791] Output: Audio data sent to the server

[1792] Step 2:

[1793] Converting audio data to text

[1794] The server converts the received voice data into text data using a voice recognition library.

[1795] Specific behavior:

[1796] Input: Audio data

[1797] Data calculation: Analyzes speech using a speech recognition algorithm (speech_recognition) and converts it into text

[1798] Output: Text data

[1799] Step 3:

[1800] Text data analysis and task instructions

[1801] The server analyzes the converted text data and instructs the robot on the appropriate task.

[1802] Specific behavior:

[1803] Input: Text data

[1804] Data Computing: Using text analysis algorithms to interpret instructions and determine the tasks to be performed

[1805] Output: Instructions to the robot to execute the task

[1806] Step 4:

[1807] Executing a task

[1808] The robot performs actual operations based on task instructions received from the server, such as stopping a conveyor belt.

[1809] Specific behavior:

[1810] Input: Task execution instructions

[1811] Data processing: The robot performs physical actions according to instructions

[1812] Output: Task completion (e.g. conveyor belt stopped)

[1813] Step 5:

[1814] Generate feedback on task completion

[1815] The server confirms the completion of the task and generates the details as text data.

[1816] Specific behavior:

[1817] Input: Task execution result

[1818] Data processing: Convert task completion information into text data

[1819] Output: Text data (e.g. "Conveyor belt has stopped")

[1820] Step 6:

[1821] Text-to-speech conversion

[1822] The text data generated by the server is converted into voice data using a voice synthesis library.

[1823] Specific behavior:

[1824] Input: Text data

[1825] Data Computing: Convert text to speech using a text-to-speech algorithm (gTTS)

[1826] Output: Audio data

[1827] Step 7:

[1828] Playback and reporting of audio data

[1829] The server sends the converted voice data to the terminal and reports it to the user through the speaker.

[1830] Specific behavior:

[1831] Input: Audio data

[1832] Data calculation: The device plays the audio data received on the speaker.

[1833] Output: User confirms audio feedback

[1834] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1835] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide users with highly accurate and natural interactions.

[1836] Overall system configuration

[1837] The system mainly consists of the following components:

[1838] 1. Means of receiving audio data

[1839] 2. Generative model for converting audio data into text data

[1840] 3. Means for editing the converted text data

[1841] 4. How to save edited text data

[1842] 5. Generative model for converting text data into speech data

[1843] 6. Means for playing back converted audio data

[1844] 7. Emotion engine for recognizing user emotions

[1845] The process of converting voice data into text data

[1846] When a user uploads an audio file, the device sends the file to the server. The server sends the audio data to a generative model for speech recognition. As a result, the audio data is converted into text data. The server then sends the text data to an emotion engine to analyze the user's emotions. Text data tagged with emotion-based tags is generated, stored in a database, and sent to the user's device. The user can review the text data on their device and edit it as necessary.

[1847] Specific examples

[1848] For example, if a user uploads a one-hour meeting recording to the platform, the device will send the audio file to the server. The server will then use a generative model to convert the speech into text. During this process, the audio data is transcribed with high accuracy and text data is generated. The emotion engine will then analyze the user's emotions and assign emotion tags to them. The generated text data will then be displayed to the user, allowing them to correct any errors in specific terms or names.

[1849] Processing to edit text data

[1850] When a user edits text data, they can make corrections using a text editor on their device. Once edited, the text data is sent to the server at the user's command and saved in the database again. This data is kept for future reuse or further editing.

[1851] Specific examples

[1852] After the text is generated, the user can correct any typos or omissions that are important to the topic. For example, if a specific name or technical term is translated incorrectly, the user can correct it accurately. Once the corrections are complete, the user clicks the save button to send the corrections to the server.

[1853] The process of converting text data into audio data

[1854] If a user wants to convert part or all of the edited text data into voice data, they select the relevant section on their device and click the "Convert to voice" button. The device then sends the selected text data to the server, which then sends the data to the generative model for voice synthesis. The generated voice data is generated by the emotion engine in a way that reflects the user's emotions, and is then sent to the user's device and played back.

[1855] Specific examples

[1856] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[1857] This concludes the detailed description of the embodiment of the present invention. This system allows users to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[1858] The processing flow will be explained below.

[1859] Converting speech to text and performing sentiment analysis

[1860] Step 1:

[1861] The user clicks on the upload button on the platform to upload the audio file.

[1862] Step 2:

[1863] The device reads the audio file selected by the user and prepares it for transmission to the server.

[1864] Step 3:

[1865] The device sends the audio file to the server as an HTTP POST request.

[1866] Step 4:

[1867] The server checks the received audio file and temporarily saves it.

[1868] Step 5:

[1869] The server sends the audio file to the speech recognition generation model, which starts the process of converting the audio data into text data.

[1870] Step 6:

[1871] The generative model analyzes the audio data, generates text data, and returns it to the server.

[1872] Step 7:

[1873] The server sends the generated text data to the emotion engine.

[1874] Step 8:

[1875] The emotion engine analyzes the text data and tags it with the user's emotions.

[1876] Step 9:

[1877] The server stores emotion-tagged text data in a database and sends it to the user's device.

[1878] Step 10:

[1879] The terminal displays the acquired text data to the user.

[1880] Text editing process

[1881] Step 1:

[1882] The user uses a text editor on the platform to perform editing operations on the text data.

[1883] Step 2:

[1884] The user clicks the save button to save the text data that he has edited.

[1885] Step 3:

[1886] The terminal prepares to send the edited text data to the server.

[1887] Step 4:

[1888] The device sends the edited text data to the server as an HTTP POST request.

[1889] Step 5:

[1890] The server checks the received edited text data and saves it in the database.

[1891] Converting text to speech and reflecting emotions

[1892] Step 1:

[1893] The user selects part or all of the edited text data and clicks a button to convert it into speech.

[1894] Step 2:

[1895] The terminal prepares to send the selected text data to the server.

[1896] Step 3:

[1897] The terminal sends the selected text data to the server as an HTTP POST request.

[1898] Step 4:

[1899] The server checks the received text data and sends it to the speech synthesis generation model.

[1900] Step 5:

[1901] The generative model analyzes the text data, generates audio data, and returns it to the server.

[1902] Step 6:

[1903] The server sends the generated voice data to the emotion engine, where it processes the data to reflect the emotion.

[1904] Step 7:

[1905] The emotion engine adds emotion information to the voice data, and the final voice data is sent to the server.

[1906] Step 8:

[1907] The server transmits the voice data that reflects the emotion to the user's terminal.

[1908] Step 9:

[1909] The device prepares to play the received audio data.

[1910] Step 10:

[1911] The terminal plays the audio data to the user.

[1912] Example 2

[1913] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1914] Conventional systems that convert voice data into text data have limitations in the accuracy of voice recognition and the naturalness of synthesized voice, and one problem is that it is particularly difficult to reflect the user's emotions. This makes it difficult for users to communicate naturally, including their emotions. The present invention aims to solve these problems and realize highly accurate conversion of voice data into text and generation of voice data that reflects emotions.

[1915] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1916] In this invention, the server includes means for receiving voice data, means for using a generative model to convert voice data into text data, means for performing emotion recognition on the converted text data, means for saving emotion-tagged text data, means for saving edited text data, means for using a generative model to convert text data into voice data, and means for playing back the converted voice data. This enables a user to convert voice data into text, edit text data, and generate voice data that reflects emotions with high accuracy and in a natural way.

[1917] "Audio data" refers to data in which audio is recorded in digital format.

[1918] "Text data" refers to data obtained by converting voice data into character information.

[1919] A "generative model" is a machine learning model used to convert voice data to text data or vice versa.

[1920] "Emotion recognition" is a technology that analyzes and identifies a user's emotions from text data and voice data.

[1921] An "emotion tag" is a tag that includes information about an emotion that is assigned to text data by emotion recognition.

[1922] A "server" is a computer system that receives voice data, converts it, recognizes emotions, stores it, and transmits the data.

[1923] A "terminal" is a device that is operated by a user and is used to send and receive voice data and text data.

[1924] "Real time" refers to a processing format in which the process from receiving voice data to generating text data is carried out without delay.

[1925] "Speech synthesis" is a technology that generates synthetic speech based on text data.

[1926] The present invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. This system integrates elements of voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interaction. The specific operation of this system is as follows.

[1927] Hardware and software used

[1928] Hardware

[1929] User's input device (terminal): PC, smartphone, tablet, etc.

[1930] Server: High-performance cloud or on-premise server

[1931] software

[1932] Speech recognition API: The general name is "Speech recognition API"

[1933] Emotion recognition engine: The generic name "emotion recognition engine"

[1934] Speech synthesis API: The general name is "Speech synthesis API"

[1935] Database: The general term "database software"

[1936] Text editor: A common text editing software installed on the device.

[1937] Overall system configuration

[1938] The system mainly consists of the following components:

[1939] 1. Means of receiving audio data

[1940] 2. Generative model for converting audio data into text data

[1941] 3. Means for editing the converted text data

[1942] 4. Means of Emotion Recognition

[1943] 5. A means of storing emotion-tagged text data

[1944] 6. How to save edited text data

[1945] 7. Generative model for converting text data into speech data

[1946] 8. Means for playing back converted audio data

[1947] Example of operation

[1948] The process of converting voice data into text data

[1949] When a user uploads an audio file, the device sends the file to the server. The server converts the audio data into text using a speech recognition API, and then sends the text to an emotion recognition engine for emotional analysis. Emotion-based tagged text data is generated, stored in a database, and sent to the user's device. The user can then review the text data on their device and edit it as necessary.

[1950] Specific prompt examples:

[1951] User: Upload a 1-hour meeting recording

[1952] Terminal: Send a file to the server

[1953] Server: Convert speech to text using speech recognition API

[1954] Server: Analyze emotions using an emotion recognition engine

[1955] Server: Generate emotion-tagged text data and store it in a database

[1956] Terminal: displays text data to the user

[1957] Processing to edit text data

[1958] The user can edit the text data using a text editor on the terminal. After editing, the text data is sent to the server at the user's instruction and saved in the database again.

[1959] Specific prompt examples:

[1960] User: Open text data in a text editor

[1961] User: Correct typos and omissions

[1962] User: Clicks the Save button to send the changes to the server

[1963] Server: Save the modified text data to the database

[1964] The process of converting text data into audio data

[1965] If a user wants to convert the edited text data into audio data, they select the relevant section on their device and click the "Convert to Audio" button. The device then sends the selected text data to the server, which then uses a speech synthesis API to generate audio data. This audio data is generated using an emotion recognition engine to reflect the user's emotions, and is then sent to the user's device and played back.

[1966] Specific prompt examples:

[1967] User: Select part of edited text data

[1968] User: Clicks the "Convert to Audio" button

[1969] Server: Sends the selected text data to the speech synthesis API

[1970] Server: Generates natural voice using emotion recognition engine

[1971] Server: Sends the generated audio data to the device

[1972] User: Press the play button to check

[1973] The above is a specific embodiment of the program processing of this system. This system allows the user to convert voice data into text with high accuracy, edit the text, and output and play back voice data that reflects emotional information.

[1974] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1975] Step 1: Receiving audio data

[1976] The user selects an audio file on the terminal and clicks the "Upload" button. The input is the audio file selected by the user. The terminal sends this audio file to the server. The output is the audio file sent to the server.

[1977] Step 2: Save the audio data to the server

[1978] The server saves the received audio file in a specific folder. The received audio file is the input, and the saved audio file is the output. At this time, the terminal displays the file transmission status on the screen.

[1979] Step 3: Converting audio data to text data

[1980] The server sends the saved voice data to a voice recognition API (commonly known as a "voice recognition API"). The voice data is input, and the API analyzes it to generate text data. The text data is obtained as output. Specifically, the API returns the text data "We discussed this project."

[1981] Step 4: Emotion Recognition

[1982] The server sends the generated text data to an emotion recognition engine (commonly known as an "emotion recognition engine"). The engine receives text data as input and identifies the user's emotion from the text data. The output is text data tagged with an emotion. Specifically, the emotion engine assigns a tag of "joy."

[1983] Step 5: Saving emotion-tagged text data

[1984] The server stores emotion-tagged text data in a database.,The input is emotion-tagged text data, and the output is the,text data stored in the database.

[1985] Step 6: Sending text data to the user

[1986] The server sends emotion-tagged text data stored in the database to the user's device. The input is text data from the database, and the output is text data sent to the user's device. The user can view this data.

[1987] Step 7: Editing text data

[1988] The user opens the received text data using a text editor on the terminal and makes the necessary corrections. The input is the transmitted text data, and the user performs editing tasks such as correcting typos and omissions. The edited text data is obtained as output. In concrete terms, the user changes "project" to "plan."

[1989] Step 8: Save the edited text data to the server

[1990] The user completes editing the text and clicks the "Save" button. The terminal sends the edited text data to the server. The input is the edited text data, and the server saves it in the database. The output is the corrected text data saved in the database.

[1991] Step 9: Converting text data to audio data

[1992] The user selects the edited text data and clicks the "Convert to speech" button. The device sends the selected text data to the server. The selected text data is input, and the server sends it to a speech synthesis API (commonly known as a "speech synthesis API") to generate speech data. The generated speech data is obtained as output.

[1993] Step 10: Send and play the generated audio data to the user

[1994] The server sends the generated voice data to the user's device. The generated voice data is input, and the voice data sent to the device is obtained as output. The user presses the play button to check the voice. Specifically, the user clicks the play button, and the text is played back in a natural voice.

[1995] (Application example 2)

[1996] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1997] Conventional food delivery services have problems that can easily lead to dissatisfaction due to misreading of order details and lack of emotional response. In particular, when ordering by voice, when users provide emotional feedback, the emotional information is not properly conveyed, making it difficult to improve service quality. Therefore, a system is needed that can accurately convert users' voice input into text and appropriately reflect emotional information, thereby deepening mutual understanding between service providers and users and improving the quality of customer support.

[1998] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1999] In this invention, the server includes means for receiving voice data, means for using a generative model to convert the voice data into text data, and an emotion engine for recognizing a user's emotion and reflecting the emotion information in the text data and voice data. This allows the user's voice input to be converted into text data with high accuracy, and makes it possible to appropriately notify the service provider of the user's emotion information.

[2000] 1. "Voice Data" means a digitized recording of a user's speech.

[2001] 2. "Text data" means digital information that has been converted from audio data into textual information.

[2002] 3. A "generative model" is a model that has been trained using machine learning or AI techniques to perform a specific task.

[2003] 4. "Means" are the methods of equipment or software used to achieve a particular purpose.

[2004] 5. "Emotion engine" is a technology that analyzes a user's voice and text data to recognize and tag emotions.

[2005] 6. "Customer Support Team" refers to the department within the service provider that handles user inquiries and resolves problems.

[2006] 7. "Notice" means a communication that conveys information to a specific recipient.

[2007] 8. "Service Provider" means a company or organization that provides food delivery or other services to Users.

[2008] 9. "Real time" means that input and output occur immediately without delay.

[2009] 10. "Speech synthesis function" refers to technology that converts text data into voice data and plays it back as natural-sounding voice.

[2010] MODE FOR CARRYING OUT THE INVENTION

[2011] This invention is a system that not only converts voice data into text data and then converts the text data back into voice data, but also recognizes the user's emotions and reflects that information in the text and voice. The system integrates voice recognition technology, voice synthesis technology, and emotion recognition technology to provide the user with highly accurate and natural interactions.

[2012] Overall system configuration

[2013] The system mainly consists of the following components:

[2014] 1. Means of receiving audio data

[2015] 2. Using generative models to convert audio data into text

[2016] 3. Means for editing the converted text data

[2017] 4. How to save edited text data

[2018] 5. Using generative models to convert text data into audio data

[2019] 6. Means for playing back converted audio data

[2020] 7. An emotion engine that recognizes user emotions and reflects that emotional information in text and voice data.

[2021] 8. Means for recognizing emotions based on user voice input and sending notifications containing emotion information to service providers

[2022] The process of converting voice data into text data

[2023] When a user speaks, the device sends this voice data to the server. The server then sends the voice data to a generative model for speech recognition. The generated text data is analyzed by an emotion engine to analyze the user's emotional information, and an emotion tag is added to the text data. This text data is then stored in a database and sent to the user's device.

[2024] Specific examples

[2025] For example, if a user reports by voice that "my order is late and I'm frustrated," this voice data is sent by the device to the server. The server uses a generative model to convert the speech into text, and in the process, the emotion engine recognizes the emotion "frustrated." Then, emotion-tagged text data is generated, stored in a database, and sent to the user's device.

[2026] Processing to edit text data

[2027] When a user edits text data, the user can make corrections using a text editor on the device. Once edited, the text data is sent to the server at the user's instruction and saved in the database again.

[2028] Specific examples

[2029] The user can check the generated text data and correct any typos, omissions, or translation errors. For example, it is possible to change "The order will not arrive quickly" to "The order is delayed." Once the corrections are complete, the user clicks the save button and sends the corrections to the server.

[2030] The process of converting text data into audio data

[2031] If a user wants to convert part or all of the edited text data into speech data, they select the relevant part on their device and click the "Convert to speech" button. The device then sends the selected text data to the server, which then sends the data to the generative model for speech synthesis.

[2032] Specific examples

[2033] When a user wants to hear the audio of the meeting summary after editing, they select the text summarizing the key points and click the "Convert to Speech" button. The server sends this text data to a generative model, which uses an emotion engine to generate natural-sounding speech. This audio data is then sent to the device, where the user can press the play button to hear it.

[2034] Hardware and software used

[2035] The hardware used is a cloud server (e.g., Amazon EC2) for high-speed processing and managing large amounts of data, and a smartphone for user interaction. The software used is a generative model for speech recognition and emotion recognition using PyTorch and Transformers.

[2036] Prompt Sentence Examples

[2037] If a user says, "I'd like to order a pizza and salad," create a program that takes the voice data, converts it to text, recognizes the sentiment, and notifies your customer support team.

[2038] As described above, this system can convert voice data into text with high accuracy, add emotional information, and output it as voice data, thereby improving the quality of customer support for food delivery services.

[2039] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2040] Step 1:

[2041] The user inputs voice orders or inquiries into the terminal. The input voice data is digitized by the terminal and sent to the server, where it is captured into the system.

[2042] Input: User's voice data

[2043] Output: The digitized audio data is sent to the server.

[2044] Step 2:

[2045] The server sends the received speech data to a generative model (speech recognition model) to convert the speech into text data. High-accuracy speech recognition is achieved using models such as PyTorch and Wav2Vec2. The generated text data is then sent to the next processing step.

[2046] Input: Digitized audio data

[2047] Output: Converted text data

[2048] Step 3:

[2049] The server sends the converted text data to the emotion engine for emotion analysis. The emotion engine analyzes the text data, recognizes the user's emotional information, and tags it. The text data with the added emotional information is stored in a database and sent to the user's device.

[2050] Input: Converted text data

[2051] Output: Text data with emotional information added

[2052] Step 4:

[2053] The user can use a text editor on the device to check the generated text data and edit it as necessary. After editing, the text data is sent from the device to the server and saved in the database.

[2054] Input: Text data with emotional information added

[2055] Output: Edited text data

[2056] Step 5:

[2057] When a user wants to convert edited text data into voice data, they select the relevant part on their device and issue a command to convert it into voice. The device then sends the selected text data to the server, which then generates voice data using a generative model (voice synthesis model).

[2058] Input: Edited text data

[2059] Output: Generated audio data

[2060] Step 6:

[2061] The generated voice data is generated in a way that reflects the user's emotional information through the emotion engine. The voice data is sent to the device, and the user can listen to the voice data by pressing the play button.

[2062] Input: Generated audio data

[2063] Output: Audio data sent to the user's device

[2064] Step 7:

[2065] Based on the user's voice input and edits, a means for sending notifications containing emotional information to the service provider is activated, for example, a notification such as "The user is unhappy with the order" is sent to the customer support team in real time.

[2066] Input: Voice input or edited data

[2067] Output: A notification containing emotion information is sent to the service provider.

[2068] The above are the specific processing steps of the system of the present invention. This configuration enables the user's voice input to be converted into text with high accuracy, emotional information to be appropriately reflected, and the quality of customer support to be improved.

[2069] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2070] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2071] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2072] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2073] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2074] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2075] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2076] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2077] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2078] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2079] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2080] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2081] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2082] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2083] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2084] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2085] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2086] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2087] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2088] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2089] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2090] The following is further disclosed regarding the above embodiment.

[2091] (Claim 1)

[2092] means for receiving audio data;

[2093] a means for utilizing a generative model to convert speech data into text data;

[2094] means for editing the converted text data;

[2095] a means for saving the edited text data;

[2096] a means for utilizing a generative model to convert text data into speech data;

[2097] means for playing the converted audio data;

[2098] A system including:

[2099] (Claim 2)

[2100] 10. The system of claim 1, wherein the system receives voice data in real time and converts it into text data.

[2101] (Claim 3)

[2102] 10. The system of claim 1, further comprising a speech synthesis function for reading text data in a natural voice.

[2103] "Example 1"

[2104] (Claim 1)

[2105] a means for a user to upload an audio file;

[2106] a means for the terminal to transmit the audio file to a server;

[2107] A means for the server to convert the voice data into text data using a generative AI model;

[2108] a means for a user to view and edit the text data on the terminal;

[2109] A means for transmitting the edited text data by the terminal to a server;

[2110] a means for the server to store the edited text data;

[2111] A means for the server to convert the edited text data into voice data using a generative AI model;

[2112] means for the terminal to play back the generated audio data;

[2113] A system including:

[2114] (Claim 2)

[2115] 10. The system of claim 1, wherein processing is performed on the audio data in real time.

[2116] (Claim 3)

[2117] 10. The system of claim 1, further comprising means for converting text data into speech data in a natural voice using a generative AI model.

[2118] "Application Example 1"

[2119] (Claim 1)

[2120] means for receiving audio data;

[2121] a means for utilizing a generative model to convert speech data into text data;

[2122] means for editing the converted text data;

[2123] a means for saving the edited text data;

[2124] a means for utilizing a generative model to convert text data into speech data;

[2125] means for playing the converted audio data;

[2126] A means to convert real-time voice instructions into text and analyze and save it,

[2127] means for outputting the results of the analysis of the text data as an audio report;

[2128] A system including:

[2129] (Claim 2)

[2130] 10. The system of claim 1, further comprising: receiving voice data in real time; converting the voice data into text data; and analyzing the text data to perform a task.

[2131] (Claim 3)

[2132] 2. The system according to claim 1, further comprising a voice synthesis function for reading out the results of the analysis of the text data in a natural voice.

[2133] "Example 2: Combining Emotion Engines"

[2134] (Claim 1)

[2135] means for receiving audio data;

[2136] a means for utilizing a generative model to convert speech data into text data;

[2137] means for performing emotion recognition on the converted text data;

[2138] a means for storing emotion-tagged text data;

[2139] a means for saving the edited text data;

[2140] a means for utilizing a generative model to convert text data into speech data;

[2141] means for playing the converted audio data;

[2142] A system including:

[2143] (Claim 2)

[2144] 10. The system of claim 1, further comprising: receiving speech data in real time; converting the speech data into text data; and applying emotion recognition to the text data.

[2145] (Claim 3)

[2146] 10. The system according to claim 1, further comprising a voice synthesis function for reading out text data in a natural voice that reflects emotional information.

[2147] "Application example 2 when combining emotion engines"

[2148] (Claim 1)

[2149] means for receiving audio data;

[2150] a means for utilizing a generative model to convert speech data into text data;

[2151] means for editing the converted text data;

[2152] a means for saving the edited text data;

[2153] a means for utilizing a generative model to convert text data into speech data;

[2154] means for playing the converted audio data;

[2155] an emotion engine that recognizes the user's emotion and reflects the emotion information in the text data and the voice data;

[2156] means for recognizing emotions based on a user's voice input and sending a notification including the emotion information to a service provider;

[2157] A system including:

[2158] (Claim 2)

[2159] 10. The system of claim 1, wherein the system receives voice data in real time and converts it into text data.

[2160] (Claim 3)

[2161] 2. The system according to claim 1, further comprising a voice synthesis function for reading out text data in a natural voice and a function for reflecting the user's emotional information. [Explanation of symbols]

[2162] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving audio data; a means for utilizing a generative model to convert speech data into text data; means for editing the converted text data; a means for saving the edited text data; a means for utilizing a generative model to convert text data into speech data; means for playing the converted audio data; A system including:

2. 10. The system of claim 1, wherein the system receives voice data in real time and converts it into text data.

3. 10. The system of claim 1, further comprising a voice synthesis function for reading text data in a natural voice.

Images