system

Abstract

We provide the system. [Solution] A means for receiving voice input, performing preprocessing, and converting the voice into text using speech recognition technology, A means of translating text converted using a translation model into a specified language, A means of converting translated text into audio data using speech synthesis technology, Means for providing the audio data to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional translation systems, it is necessary to process information in different formats such as voice, text, and images separately, and users had the hassle of using multiple applications. Also, in systems that provide high-precision real-time translation, there were technical problems in accurately understanding the context and realizing natural expressions. As a result, it has been required to achieve smooth and accurate communication in various scenarios.

Means for Solving the Problems

[0005] The present invention provides a system that includes means for receiving voice input, performing preprocessing, and converting the voice to text using speech recognition technology. It then includes means for translating the converted text into a specified language using a translation model, and further includes means for converting the translated text into voice data using speech synthesis technology and providing it to the user. The system also includes translation means based on text input and character extraction from image data, and is configured to process multiple information formats centrally, thereby providing a consistent user experience.

[0006] "Voice input" refers to information provided by the user in the form of voice, and is the initial data used to convert that information into a format that can be processed within the system.

[0007] "Preprocessing" refers to a series of processes performed to convert audio or text data into a parseable format, and includes noise reduction and format normalization.

[0008] "Speech recognition technology" refers to the technology used to analyze audio data and extract its content as text; it is the process of converting spoken language into written characters.

[0009] "Means of converting to text" refers to methods and technologies for converting audio data into text data, and utilizes speech recognition technology.

[0010] A "translation model" is a model used to translate content written in one language into another language, and is usually trained using machine learning.

[0011] "Means of translating into a specified language" refers to methods and technologies for converting input text data into another language, and is implemented using a translation model.

[0012] "Speech synthesis technology" is a technology that converts text data into speech data and artificially generates speech, and is based on TTS (text-to-speech) technology.

[0013] "Means of providing to users" refers to methods or technologies for transmitting processed data to end users and making the results available to them.

[0014] "Text input" refers to the act of a user directly entering or providing text information to a system, using a keyboard or touchscreen.

[0015] "Image data" refers to data containing information represented as a still image, and this data is used to extract textual information.

[0016] Optical Character Recognition (OCR) is a technology used to extract text information from image data.

[0017] "Means for extracting textual information" refers to methods and techniques for analyzing characters contained within an image and extracting them as text. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the language used in the following description will be described.

[0021] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] As an embodiment of the present invention, an example is described in which an integrated platform for processing voice, text, and image inputs is established in a system that provides real-time, high-precision translation and interpretation. This system centrally processes various input methods and utilizes a generative AI model to provide multilingual translations.

[0040] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and voice normalization. Next, it converts the voice into text using speech recognition technology and sends it to the server. The server translates the received text into the specified language using a translation model. Then, it sends the translation result back to the device as audio data using speech synthesis technology. The device plays this audio data and provides it to the user.

[0041] In the case of text input, the user enters text into the terminal and sends it to the server. The server translates the entered text data into the specified language using a translation model. The translated result is sent back to the terminal and displayed on the terminal. The user can check the translation result on the screen.

[0042] In image translation, the user takes or selects an image using their device and sends it to the server. The device first converts the image data into text information using OCR technology. The server translates the extracted text into the specified language using a translation model and sends the result to the device. The device overlays the translated result onto the original image, allowing the user to visually confirm the text before and after translation.

[0043] For example, if a user wants to take a picture of a French menu and translate it into English, the image captured by the device is converted into text information using OCR. The server translates this text into English, and the device overlays the translation result onto the menu image for the user to see. The user can then understand the menu content based on this visual information.

[0044] This integrated system eliminates the complexities of traditional methods, allowing users to seamlessly process various inputs such as voice, text, and images, and obtain translation results quickly. This invention serves as a solution for removing language barriers in a variety of scenarios.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The user inputs voice into the device, and the device captures that voice using its microphone. The device preprocesses the voice data through noise reduction and normalization.

[0048] Step 2:

[0049] The device converts pre-processed audio data into text data using speech recognition technology. This conversion allows it to understand spoken language as written text.

[0050] Step 3:

[0051] The terminal sends the converted text data to the server. A secure data transmission protocol is used to send the data.

[0052] Step 4:

[0053] The server inputs the received text data into a translation model and translates it into the specified language. This step ensures an appropriate translation that takes context and nuance into account.

[0054] Step 5:

[0055] The server converts the translation results into audio data using text-to-speech (TTS) technology. This process generates the translation results as natural-sounding speech.

[0056] Step 6:

[0057] The server sends the generated audio data to the terminal.

[0058] Step 7:

[0059] The device plays the received audio data through its speaker, conveying the translated content to the user.

[0060] Step 8:

[0061] Users can view and use the translation results through their devices.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] Translation systems that handle diverse input formats require individual process management for each format, leading to increased complexity and processing time for users. Furthermore, there is a demand for improved translation accuracy between different languages ​​and real-time processing.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for acquiring audio data, pre-processing it with data processing technology, and converting it into text data using speech recognition technology; means for translating the text data into a target language using a generative AI model; and means for converting the translated text data into a digital audio format and outputting it as audio using presentation technology. This enables centralized processing of various input formats and allows for highly accurate and real-time multilingual translation.

[0067] "Audio data" refers to sound information that is electronically captured, such as a user's speech, and represented in a digital format.

[0068] "Data processing technology" refers to operations such as filtering and noise reduction to adjust raw input data into a usable state.

[0069] "Speech recognition technology" is a technology for converting speech data into text format, and includes processing such as phonological analysis and language models.

[0070] "Text data" refers to character information expressed in a digital format, and is used for purposes such as translation and storage.

[0071] A "generative AI model" is an artificial intelligence system trained on a large dataset, capable of multilingual translation and creative text generation.

[0072] The "target language" is the language that is required as the translation result, to which the text data is converted during the translation process.

[0073] A "digital audio format" refers to an electronically represented format that converts text data into speech, and is audio data that can be played back on speakers or other devices.

[0074] "Presentation technology" refers to interfaces and media used to transmit converted information to users, and is a method of providing information through senses such as sight and hearing.

[0075] An "integrated platform" is a foundational system that centrally manages different input formats and processing technologies to provide an efficient and seamless user experience.

[0076] This invention relates to a system that integrates and processes multiple input formats to provide rapid multilingual translation. This system supports voice input, text input, and image input, and provides users with highly accurate translation results through processes appropriate to each format.

[0077] If the user selects voice input, the device captures voice data using its built-in microphone. The device then processes the voice data, performing noise reduction and normalization, and converts it into text data using speech recognition technology. A common speech recognition library is used for this process. The converted text data is sent to a server, which translates it into the target language using a generative AI model. For example, if the user inputs "Bonjour" by voice, the system will respond with "Hello" by voice.

[0078] In the case of text input, the user directly enters text into the device and sends it to the server. The server uses a generative AI model to translate the input text and sends the translated text back to the device. The user can then view the translated text on the screen.

[0079] For image input, the user either takes a picture with the device's camera or selects an existing image. The device uses OCR technology to extract text information from the image and generates text data. This data is sent to a server and translated by a generative AI model. The device then overlays the translated text onto the original image, allowing the user to visually understand the image. For example, in a scenario where a user takes a picture of a French sign and translates it into English, the word "Café" extracted from the image taken by the user will be displayed as "Cafe".

[0080] An example of a prompt would be, "Translate the French text in this image into English." This clearly communicates the translation request to the generative AI model and provides instructions for performing an appropriate translation.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] Users can input information into the device in the form of voice, text, or images. For voice input, the user speaks into the microphone, and the device captures this as audio data. For text input, the user uses the device's keyboard to type. For image input, the user either takes a photo with the camera or selects an existing image.

[0084] Step 2:

[0085] The device applies a noise reduction filter to the captured audio data. This data processing removes external noise. Normalization adjusts the volume to maintain a constant amplitude in the audio signal. The input is raw audio data, and the output is de-noised audio data.

[0086] Step 3:

[0087] The device converts pre-processed audio data into text data using speech recognition software. For example, it converts the user's utterance "Hello, how are you?" into the text "Hello, how are you?". The input is devoiced audio data, and the output is the corresponding text.

[0088] Step 4:

[0089] When using image input, the device uses OCR technology to recognize characters within the image and extract text information. For example, from an image containing the word "Café," the text "Café" will be extracted. The input is image data, and the output is the text information within the image.

[0090] Step 5:

[0091] Text data (including data extracted by speech recognition or OCR) is sent from the terminal to the server. The input is the text data generated by the terminal, and the output is the data sent to the server.

[0092] Step 6:

[0093] The server translates the received text data using a generative AI model. The generative AI model is trained on a large language dataset and, as an example, translates "Bonjour" to "Hello". The input is the text data entered by the user, and the output is the translated text data.

[0094] Step 7:

[0095] The server sends the translation result to the terminal. The input is the translated text data, and the output is the translated data sent to the terminal.

[0096] Step 8:

[0097] The device uses speech synthesis software to convert translated text data into audio data and play it through the speaker. For example, "Hello" will be played in English. In the case of text input, the device displays the translation result on the screen. In the case of image input, the translation result is overlaid on the original image and presented to the user. The input is translated text data, and the output is either audio data or displayed data.

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] Enabling smooth communication between different languages ​​is a crucial challenge in today's increasingly globalized society. Language barriers can be a major obstacle, especially during travel and cross-cultural exchange. Traditional translation methods require separate processing for audio, text, and image inputs, making them cumbersome for users. Therefore, there is a need for a new system that integrates these inputs and provides real-time translation results.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes means for receiving voice input and converting it into text using speech recognition technology, means for translating the text into a specified language using a translation model and converting it into voice data using speech synthesis technology, and means for processing image data, extracting character information using optical character recognition technology, and displaying the translated results on the image. This enables integrated processing of voice, text, and image information, and facilitates smooth communication between multiple languages.

[0103] "Voice input" is the process used to capture voice information into a digital device.

[0104] "Preprocessing" is the process of removing noise and unwanted parts from audio or text data so that subsequent processing can be performed more effectively.

[0105] "Speech recognition technology" is a technology that analyzes speech data and converts it into text information.

[0106] "Text" refers to a form of data expressed as character information.

[0107] A "translation model" is an algorithm or program used to convert input language data into another specified language.

[0108] "Speech synthesis technology" is a technology that converts text data into speech data and reproduces it in a way that sounds like human speech.

[0109] "Audio data" refers to sound information expressed in digital format.

[0110] A "user" is an individual or group that uses a system or service.

[0111] "Image data" refers to data that represents visual information in a digital format.

[0112] "Optical character recognition technology" is a technology that identifies character information from images and converts it into digital character data.

[0113] "Character information" refers to data content that consists of characters as its constituent elements.

[0114] "Display" refers to the process or means of presenting information visually.

[0115] "Multilingual travel support" refers to functions or services that enable travelers to understand and utilize information, overcoming language barriers.

[0116] The system that realizes this invention integrates speech recognition technology, optical character recognition technology, translation models, and speech synthesis technology to provide smooth multilingual translation.

[0117] The server receives audio, text, or image data over the network. For audio data, the server uses speech recognition technology to convert the speech to text. This process utilizes libraries such as Librosa and PyDub to denoise and normalize the audio. Then, a generative AI model is used to translate the text into the specified language. The translated text is then converted back into audio data using a speech synthesis service like Amazon Polly and sent to the user's device.

[0118] For image data, the server first extracts text information from the image using Tesseract OCR technology. The extracted text is then translated into the target language using a generative AI model. The translation result is overlaid on the original image using the OpenCV library and presented to the user visually.

[0119] Users can obtain multilingual travel assistance using smartphones and robots. For example, if a user wants to translate a French menu into English, an image taken with the device is converted into text information using OCR technology and then translated by a generative AI model. An example of a prompt would be, "Please use voice input to translate the French menu into English."

[0120] In this way, the system centrally processes diverse information inputs such as voice, text, and images, providing users with the ability to obtain information beyond language barriers.

[0121] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0122] Step 1:

[0123] The device accepts voice input from the user. After acquiring the voice data, Librosa is used to perform noise reduction and normalization of the speech. This preprocessing results in data that is more suitable for accurate speech recognition.

[0124] Step 2:

[0125] The terminal sends pre-processed audio data to the server. The server uses speech recognition technology to convert the audio data into text data. Here, the audio data is transformed into text format. This text data is then sent to the next stage.

[0126] Step 3:

[0127] The server feeds text data into a generative AI model, which then translates it into the specified target language. The translation is performed using prompts, and the text in the target language is output.

[0128] Step 4:

[0129] The text data translated on the server is then fed back into speech synthesis technology and converted back into speech data. Natural-sounding speech is generated using speech synthesis services such as Amazon Polly.

[0130] Step 5:

[0131] The generated audio data is sent from the server to the terminal. The terminal plays this audio data and provides it to the user. This allows the user to obtain audio information in their specified language.

[0132] Step 6:

[0133] When a user uses an image, the device sends the captured image data to the server. The server uses Tesseract OCR to extract text from the image and obtains this text data.

[0134] Step 7:

[0135] The server feeds the extracted text into a generative AI model for translation into the target language. The translated text is then processed to be placed on top of the original image.

[0136] Step 8:

[0137] Using OpenCV, the translation result is overlaid onto the original image. The device provides the user with the processed image data, allowing them to visually understand the information.

[0138] The above is a detailed processing flow of the system that implements the application example.

[0139] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0140] As an embodiment of the present invention, an example of providing a highly accurate real-time translation system that takes user emotions into consideration will be described. This system processes voice, text, and image input, recognizes the user's emotions using an emotion engine, and reflects the results in the translation.

[0141] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and normalization. Next, speech recognition technology is used to convert the voice into text, and this text information is passed to the generating AI and emotion engine. The emotion engine estimates the user's emotions from the tone of voice and nuances of speech. Using this emotion information, the server translates the text into the specified language using a translation model and makes corrections to appropriate wording and tone. The translation result is then converted into audio data through speech synthesis technology and sent to the device. The device plays this audio and provides it to the user.

[0142] In the case of text input, the user directly enters text into the device and sends it to the server. The server analyzes the sentiment of the text through its sentiment engine and passes the results to a translation model. Based on this sentiment information, it generates a translation result incorporating appropriate sentimental expressions and sends it back to the device. The device then displays the translated text on the screen and provides it to the user.

[0143] In image translation, the user uses their device to take or select an image and sends it to the server. The device uses OCR technology to extract text information from the image and sends this text to the server. The server uses an emotion engine to analyze the facial expressions of people and the context of the subjects in the image. This analysis is then incorporated into a translation model to produce an emotionally appropriate translation. The result is sent to the device and displayed as an overlay on the original image.

[0144] For example, if a user wants to translate an image sent by a friend, the emotion engine recognizes positive elements in the image, such as the smiles of the people in the image or the background. The server then uses this emotion information to perform the translation, generating and providing a more friendly and lighthearted translation. This allows the user to understand the context more deeply and achieve richer communication.

[0145] This system goes beyond simple language conversion, providing natural and accurate translations that reflect the user's emotions, thereby improving the quality of communication.

[0146] The following describes the processing flow.

[0147] Step 1:

[0148] The user inputs voice into the device, and the device captures that voice using its microphone. The device then performs preprocessing on this audio data, such as noise reduction and normalization.

[0149] Step 2:

[0150] The device uses speech recognition technology to convert pre-processed audio data into text data. This text data forms the basis for the translation process.

[0151] Step 3:

[0152] The terminal sends the converted text data to the server. At this time, the emotion engine extracts emotion data based on the tone and nuances unique to the voice.

[0153] Step 4:

[0154] The server uses an emotion engine to analyze emotional information obtained from the received text and determine the estimated emotional state of the user.

[0155] Step 5:

[0156] The server inputs text into a translation model, which then translates it into the specified language, taking into account the emotional information contained within. At this stage, the wording and tone are appropriately adjusted.

[0157] Step 6:

[0158] The server converts the translated text into speech data using speech synthesis technology, generating speech that reflects emotional expression.

[0159] Step 7:

[0160] The server sends the generated audio data to the terminal.

[0161] Step 8:

[0162] The device plays back the received audio data and provides the user with a translation result that reflects the emotions.

[0163] Step 9:

[0164] Users can view translated content through their devices and engage in natural communication.

[0165] (Example 2)

[0166] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0167] Traditional translation systems simply convert language, making it difficult to provide natural and accurate translations that take into account the user's emotions and context. Furthermore, there is a need to support different input formats such as audio, text, and images, while providing emotionally reflective translation results in real time.

[0168] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0169] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into text information using speech recognition technology; means for analyzing the emotions of the converted text information using emotion analysis technology; and means for converting the text information into a specified language using a translation model, taking the emotional information into consideration, and modifying the wording and tone to match the emotions. This makes it possible to provide translations that reflect the user's emotions in real time using different input formats.

[0170] "Voice input" is a method by which users provide information to a system through speaking.

[0171] "Preprocessing" refers to the initial processing performed to prepare input data for analysis and processing.

[0172] "Speech recognition technology" is a technology that converts speech into text data that can be understood by computers and other devices.

[0173] "Text information" refers to data represented as a string of characters, and is information used for language processing.

[0174] "Emotional analysis technology" is a technology that analyzes and extracts emotional states from text and audio data.

[0175] "Emotional information" refers to data that indicates the user's emotional state and is information that influences translation and other processing.

[0176] A "translation model" is an algorithm used to convert text from one language to another.

[0177] "Language choice" refers to the selection and tone of language used in communication.

[0178] "Intonation" refers to elements that represent the phonetic intonation and atmosphere of spoken language and written text.

[0179] "Speech synthesis technology" is a technology that converts text data into speech data.

[0180] "Audio data" refers to audio information expressed in digital format.

[0181] A "user" is a person or group that uses this system.

[0182] "Real-time" refers to processing that is performed with minimal delay and in an immediate manner.

[0183] "Different input formats" is a concept that includes a variety of data forms such as audio, text, and images.

[0184] This invention provides a highly accurate real-time translation system that takes user emotions into consideration. This system processes different input formats, namely voice, text, and images, estimates the user's emotions using sentiment analysis technology, and reflects the results in the translation.

[0185] When a user provides voice input, they provide their voice through the device's microphone. The device captures this voice data and performs noise reduction and normalization using libraries such as "LibROSA". Next, speech recognition technology such as "Google® Speech-to-Text" converts the voice information into text information. This converted text information is sent to a server, which analyzes the emotions using sentiment analysis technology such as "IBM Watson® Tone Analyzer". Having obtained the emotion information, the server uses a translation model (e.g., DeepL API) to convert the text information into the specified language, making adjustments to the wording and tone according to the emotions. Subsequently, speech synthesis technology (e.g., "Amazon Polly") converts the text information back into voice data and sends it back to the device. The device provides this voice data to the user through its speaker.

[0186] In the case of text input, the user enters the content they wish to convert into the text input field on their device. The entered text information is sent to the server, which analyzes the sentiment of the text using sentiment analysis technology. Taking the sentiment information into account, the server uses a translation model to convert the text into appropriate language and expression, and sends the result back to the device. The device then displays the translation result on the screen and provides it to the user.

[0187] In image translation, the user uses their device to take or select an image they want to translate and sends it to the server. The server extracts text information from the image using optical character recognition (OCR) technology such as "Tesseract OCR" and evaluates the sentiment of the text and visual elements within the image using sentiment analysis technology. The server then inputs this sentiment information into a translation model and converts it into the appropriate language. The converted text is added as an overlay to the original image, and the device displays it to the user.

[0188] For example, if a user wants to translate a picture of a smiling person sent by a friend, the emotion analysis technology recognizes the smile of the subject in the image. The server interprets this as a positive emotion and reflects a friendly and approachable tone in the translation result. An example of a prompt would be, "Analyze this text with the emotion engine and provide a translation that matches the emotion."

[0189] This system goes beyond simple language conversion, enabling natural and accurate translations that reflect emotions, thereby improving the quality of communication.

[0190] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0191] Step 1:

[0192] Users input data as voice, text, or images using the device. For voice input, the user speaks through the device's microphone, and the device captures the voice data. For text input, the user directly enters text into the device's input field. For image input, the user prepares the image data by selecting the device's camera or a saved image.

[0193] Step 2:

[0194] The terminal performs noise reduction and normalization on the captured audio data. If the input data is audio, it uses a library such as "LibROSA" to perform preprocessing to convert the audio to text. This removes noise and outputs audio data that is easier to analyze.

[0195] Step 3:

[0196] The device performs speech recognition processing and converts the speech data into text information using technologies such as "Google Speech-to-Text". This conversion outputs the speech as text data that can be processed by a computer.

[0197] Step 4:

[0198] The server receives text data sent from the terminal and analyzes the sentiment of the text using sentiment analysis technology. Tools such as "IBM Watson Tone Analyzer" are used to estimate sentiment from the tone and context of the text and obtain sentiment information. This sentiment information influences the translation process.

[0199] Step 5:

[0200] The server considers the results of sentiment analysis and uses a translation model to convert the text data into the specified language. Translation models such as "DeepL API" are used to correct the wording and tone to suit the emotion. As a result of this process, the text is output as a translation that reflects the emotion.

[0201] Step 6:

[0202] The server uses speech synthesis technology to convert translated text information into audio data. Using services like "Amazon Polly," it converts the translated text into audio data and sends it to the terminal. Audio output is obtained at this stage.

[0203] Step 7:

[0204] The device receives audio data sent from the server and provides it to the user through the speaker. For text input, the translated text is displayed on the screen. For images, the translated text is displayed as an overlay and provided to the user. This process allows the user to receive translation results that are based on natural emotions.

[0205] (Application Example 2)

[0206] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0207] Current translation systems simply convert words into another language, making it difficult to provide natural translations that reflect the user's emotions. This often leads to a loss of emotional nuance in communication, particularly in multilingual households and intercultural exchanges, resulting in a decline in the quality of conversation. Conventional technologies lack mechanisms to recognize and apply emotions to translation, thus creating a need for technologies that enable more emotionally rich communication.

[0208] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0209] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into a sequence of symbols using acoustic recognition technology; means for identifying emotions and passing the estimated emotion information to a generation AI model; and means for translating the converted sequence of symbols into a specified symbol system using a translation model and adjusting the wording and tone based on the emotion information. This enables natural and accurate translation that reflects the user's emotions.

[0210] "Voice input" refers to the acoustic signals obtained when a user speaks, and it is a fundamental means for a system to receive information.

[0211] "Preprocessing" is the process of removing noise and normalizing audio and text data, and is preparatory work to improve recognition accuracy.

[0212] "Acoustic recognition technology" is a technology that analyzes audio data and generates text information from it; it is a process of converting human language into a format that computers can understand.

[0213] A "symbol sequence" is a sequence of characters or symbols that results from the processing and conversion of speech or text; it is an abstract form of information representation.

[0214] "Emotional identification" is a technology that identifies the emotional nuances behind input voice or text, and is a process for estimating the user's emotional state.

[0215] A "generative AI model" is a system that utilizes artificial intelligence to produce creative responses and outputs based on input data, and is a model designed to provide information that is appropriate to emotions and context.

[0216] A "translation model" is an algorithm or mechanism for converting text from one language to another, and it is a technology that acts as a bridge for conveying meaning between different languages.

[0217] A "designated symbolic system" refers to the target language or form of expression, and is a set of languages ​​and symbols defined in advance as the target of translation or processing.

[0218] "Speech synthesis technology" is a technology that artificially generates acoustic signals based on text data and outputs them as natural-sounding speech. It is a process that converts textual information into a form that humans can recognize by hearing.

[0219] "Audio data" refers to the digital format of sound generated by speech synthesis technology, and is the final output information provided to the user.

[0220] A "user" is a person or organization that receives services provided by the system, and is the entity that receives translation and information through interaction with the system.

[0221] As an embodiment of this invention, the system is constructed as follows. This system processes voice, text, and image input and provides real-time translation that takes emotion into account.

[0222] First, the user inputs voice into the device. The device captures the voice using its microphone and performs preprocessing such as noise reduction and normalization. Next, the device uses speech recognition technology (for example, the Google Speech-to-Text API) to convert the voice into text and sends that data to the server.

[0223] The server receives text data and uses an emotion analysis engine (for example, IBM Watson Tone Analyzer) to recognize emotions. This engine estimates the user's emotions from the tone of voice and nuances of speech, and passes that information to a generative AI model.

[0224] The generative AI model receives text to be translated based on sentiment information and uses a translation model (e.g., Google Translate API) to perform a translation that reflects natural and appropriate sentiment in the specified language. The translation result is then converted back into audio data using speech synthesis technology (e.g., Amazon Polly).

[0225] Finally, the device plays this audio data through its speaker and provides it to the user. This allows the user to receive the translated result, which reflects emotions, as audio.

[0226] As a concrete example, when a home robot conveys the gentle words of a parent who speaks Japanese to a child learning English, the voice assistant can analyze the parent's tone using an emotion engine, generate English with a positive nuance using an AI model, and provide a translation result that gives the child a sense of security.

[0227] As an example of a prompt, by inputting a request such as, "When Mom asks the robot, 'How was your homework today?', please convey the message in a gentle tone," the system can generate output corresponding to that instruction.

[0228] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0229] Step 1:

[0230] The user inputs voice into the device's microphone. The device captures the voice and performs initial preprocessing. This preprocessing applies a noise reduction filter and normalizes the audio signal, preparing it for high-quality data to be sent to the server. The input is raw audio data, and the output is normalized audio data with noise removed.

[0231] Step 2:

[0232] The device converts speech to text by sending pre-processed audio data to a speech recognition engine (e.g., Google Speech-to-Text API). The speech recognition engine analyzes the acoustic characteristics and generates corresponding string data. The input is de-noised audio data, and the output is the converted text data.

[0233] Step 3:

[0234] The terminal transfers the generated text data to the server. The server receives the text data and analyzes the sentiment information using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer). The sentiment analysis engine estimates the user's emotions from the structure and word choice of the text and outputs it as an emotion vector. The input is text data, and the output is an emotion vector.

[0235] Step 4:

[0236] The server sends emotion vectors to a generating AI model, which then passes them along with prompt text to a translation model (e.g., Google Translate API). This model considers the emotion information and translates the text into the specified language while maintaining appropriate tone and wording. The input is text data and emotion vectors, and the output is the translated text.

[0237] Step 5:

[0238] The server passes the translated text to a speech synthesis technology (e.g., Amazon Polly) to generate audio data. The speech synthesis engine synthesizes speech with natural tones and intonations that reflect emotional information, and outputs it as digital audio data. The input is the translated text, and the output is the synthesized audio data.

[0239] Step 6:

[0240] The terminal provides the user with audio data received from the server by playing it through its speaker. By listening to this, the user can receive a translation that reflects emotions. The input is synthesized audio data, and the output is the played audio.

[0241] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0242] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0243] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0244] [Second Embodiment]

[0245] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0246] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0247] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0248] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0249] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0250] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0251] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0252] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0253] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0254] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0255] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0256] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0257] As an embodiment of the present invention, an example is described in which an integrated platform for processing voice, text, and image inputs is established in a system that provides real-time, high-precision translation and interpretation. This system centrally processes various input methods and utilizes a generative AI model to provide multilingual translations.

[0258] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and voice normalization. Next, it converts the voice into text using speech recognition technology and sends it to the server. The server translates the received text into the specified language using a translation model. Then, it sends the translation result back to the device as audio data using speech synthesis technology. The device plays this audio data and provides it to the user.

[0259] In the case of text input, the user enters text into the terminal and sends it to the server. The server translates the entered text data into the specified language using a translation model. The translated result is sent back to the terminal and displayed on the terminal. The user can check the translation result on the screen.

[0260] In image translation, the user takes or selects an image using their device and sends it to the server. The device first converts the image data into text information using OCR technology. The server translates the extracted text into the specified language using a translation model and sends the result to the device. The device overlays the translated result onto the original image, allowing the user to visually confirm the text before and after translation.

[0261] For example, if a user wants to take a picture of a French menu and translate it into English, the image captured by the device is converted into text information using OCR. The server translates this text into English, and the device overlays the translation result onto the menu image for the user to see. The user can then understand the menu content based on this visual information.

[0262] This integrated system eliminates the complexities of traditional methods, allowing users to seamlessly process various inputs such as voice, text, and images, and obtain translation results quickly. This invention serves as a solution for removing language barriers in a variety of scenarios.

[0263] The following describes the processing flow.

[0264] Step 1:

[0265] The user inputs voice into the device, and the device captures that voice using its microphone. The device preprocesses the voice data through noise reduction and normalization.

[0266] Step 2:

[0267] The device converts pre-processed audio data into text data using speech recognition technology. This conversion allows it to understand spoken language as written text.

[0268] Step 3:

[0269] The terminal sends the converted text data to the server. A secure data transmission protocol is used to send the data.

[0270] Step 4:

[0271] The server inputs the received text data into a translation model and translates it into the specified language. This step ensures an appropriate translation that takes context and nuance into account.

[0272] Step 5:

[0273] The server converts the translation results into audio data using text-to-speech (TTS) technology. This process generates the translation results as natural-sounding speech.

[0274] Step 6:

[0275] The server sends the generated audio data to the terminal.

[0276] Step 7:

[0277] The device plays the received audio data through its speaker, conveying the translated content to the user.

[0278] Step 8:

[0279] The user checks and uses the translation result through the terminal.

[0280] (Example 1)

[0281] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0282] In a translation system that handles various input formats, individual process management for each format is required, and there is a problem that the complexity and processing time faced by users increase. In addition, it is required to improve the translation accuracy between different languages and realize real-time processing.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0284] In this invention, the server includes means for acquiring voice data, performing preprocessing with data processing technology, and converting it into text data using voice recognition technology; means for translating text data into a target language by utilizing a generated AI model; and means for converting the translated text data into a digital voice format and outputting the voice with presentation technology. Thereby, various input formats can be processed uniformly, and high-precision and real-time multilingual translation becomes possible.

[0285] "Voice data" is information of sound that electronically captures the user's speech or the like and is expressed in a digital format.

[0286] "Data processing technology" refers to operations such as filtering and noise removal for adjusting the input raw data into a usable state.

[0287] "Voice recognition technology" is a technology for converting voice data into text format and includes processes such as phonetic analysis and language models.

[0288] "Text data" refers to character information expressed in a digital format, and is used for purposes such as translation and storage.

[0289] A "generative AI model" is an artificial intelligence system trained on a large dataset, capable of multilingual translation and creative text generation.

[0290] The "target language" is the language that is required as the translation result, to which the text data is converted during the translation process.

[0291] A "digital audio format" refers to an electronically represented format that converts text data into speech, and is audio data that can be played back on speakers or other devices.

[0292] "Presentation technology" refers to interfaces and media used to transmit converted information to users, and is a method of providing information through senses such as sight and hearing.

[0293] An "integrated platform" is a foundational system that centrally manages different input formats and processing technologies to provide an efficient and seamless user experience.

[0294] This invention relates to a system that integrates and processes multiple input formats to provide rapid multilingual translation. This system supports voice input, text input, and image input, and provides users with highly accurate translation results through processes appropriate to each format.

[0295] If the user selects voice input, the device captures voice data using its built-in microphone. The device then processes the voice data, performing noise reduction and normalization, and converts it into text data using speech recognition technology. A common speech recognition library is used for this process. The converted text data is sent to a server, which translates it into the target language using a generative AI model. For example, if the user inputs "Bonjour" by voice, the system will respond with "Hello" by voice.

[0296] In the case of text input, the user directly enters text into the device and sends it to the server. The server uses a generative AI model to translate the input text and sends the translated text back to the device. The user can then view the translated text on the screen.

[0297] For image input, the user either takes a picture with the device's camera or selects an existing image. The device uses OCR technology to extract text information from the image and generates text data. This data is sent to a server and translated by a generative AI model. The device then overlays the translated text onto the original image, allowing the user to visually understand the image. For example, in a scenario where a user takes a picture of a French sign and translates it into English, the word "Café" extracted from the image taken by the user will be displayed as "Cafe".

[0298] An example of a prompt would be, "Translate the French text in this image into English." This clearly communicates the translation request to the generative AI model and provides instructions for performing an appropriate translation.

[0299] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0300] Step 1:

[0301] The user inputs to the terminal in the form of voice, text, or image. In the case of voice input, the user speaks towards the microphone, and the terminal captures this as voice data. For text input, the user uses the terminal's keyboard to input text. In the case of image input, the user takes a photo with the camera or selects an existing image.

[0302] Step 2:

[0303] The terminal applies a noise removal filter to the captured voice data. By this data processing, external noise is removed. In the normalization process, volume adjustment is performed to keep the amplitude of the voice signal constant. The input is raw voice data, and the output is denoised voice data.

[0304] Step 3:

[0305] The terminal converts the preprocessed voice data into text data using speech recognition software. For example, it converts the user's speech "Hello, how are you?" into the text "Hello, how are you?". The input is denoised voice data, and the output is the corresponding text.

[0306] Step 4:

[0307] In the case of image input, the terminal uses OCR technology to recognize the characters in the image and extract text information. For example, from an image containing the word "Café", the text "Café" is extracted. The input is image data, and the output is the text information in the image.

[0308] Step 5:

[0309] The text data (including the data after speech recognition or extraction by OCR) is sent from the terminal to the server. The input is the text data generated by the terminal, and the output is the data sent to the server.

[0310] Step 6:

[0311] The server translates the received text data using a generative AI model. The generative AI model is trained on a large language dataset and, as an example, translates "Bonjour" to "Hello". The input is the text data entered by the user, and the output is the translated text data.

[0312] Step 7:

[0313] The server sends the translation result to the terminal. The input is the translated text data, and the output is the translated data sent to the terminal.

[0314] Step 8:

[0315] The device uses speech synthesis software to convert translated text data into audio data and play it through the speaker. For example, "Hello" will be played in English. In the case of text input, the device displays the translation result on the screen. In the case of image input, the translation result is overlaid on the original image and presented to the user. The input is translated text data, and the output is either audio data or displayed data.

[0316] (Application Example 1)

[0317] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0318] Enabling smooth communication between different languages ​​is a crucial challenge in today's increasingly globalized society. Language barriers can be a major obstacle, especially during travel and cross-cultural exchange. Traditional translation methods require separate processing for audio, text, and image inputs, making them cumbersome for users. Therefore, there is a need for a new system that integrates these inputs and provides real-time translation results.

[0319] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0320] In this invention, the server includes means for receiving voice input and converting it into text using speech recognition technology, means for translating the text into a specified language using a translation model and converting it into voice data using speech synthesis technology, and means for processing image data, extracting character information using optical character recognition technology, and displaying the translated results on the image. This enables integrated processing of voice, text, and image information, and facilitates smooth communication between multiple languages.

[0321] "Voice input" is the process used to capture voice information into a digital device.

[0322] "Preprocessing" is the process of removing noise and unwanted parts from audio or text data so that subsequent processing can be performed more effectively.

[0323] "Speech recognition technology" is a technology that analyzes speech data and converts it into text information.

[0324] "Text" refers to a form of data expressed as character information.

[0325] A "translation model" is an algorithm or program used to convert input language data into another specified language.

[0326] "Speech synthesis technology" is a technology that converts text data into speech data and reproduces it in a way that sounds like human speech.

[0327] "Audio data" refers to sound information expressed in digital format.

[0328] A "user" is an individual or group that uses a system or service.

[0329] "Image data" refers to data that represents visual information in a digital format.

[0330] "Optical character recognition technology" is a technology that identifies character information from images and converts it into digital character data.

[0331] "Character information" refers to data content that consists of characters as its constituent elements.

[0332] "Display" refers to the process or means of presenting information visually.

[0333] "Multilingual travel support" refers to functions or services that enable travelers to understand and utilize information, overcoming language barriers.

[0334] The system that realizes this invention integrates speech recognition technology, optical character recognition technology, translation models, and speech synthesis technology to provide smooth multilingual translation.

[0335] The server receives audio, text, or image data over the network. For audio data, the server uses speech recognition technology to convert the speech to text. This process utilizes libraries such as Librosa and PyDub to denoise and normalize the audio. Then, a generative AI model is used to translate the text into the specified language. The translated text is then converted back into audio data using a speech synthesis service like Amazon Polly and sent to the user's device.

[0336] For image data, the server first extracts text information from the image using Tesseract OCR technology. The extracted text is then translated into the target language using a generative AI model. The translation result is overlaid on the original image using the OpenCV library and presented to the user visually.

[0337] Users can obtain multilingual travel assistance using smartphones and robots. For example, if a user wants to translate a French menu into English, an image taken with the device is converted into text information using OCR technology and then translated by a generative AI model. An example of a prompt would be, "Please use voice input to translate the French menu into English."

[0338] In this way, the system centrally processes diverse information inputs such as voice, text, and images, providing users with the ability to obtain information beyond language barriers.

[0339] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0340] Step 1:

[0341] The device accepts voice input from the user. After acquiring the voice data, Librosa is used to perform noise reduction and normalization of the speech. This preprocessing results in data that is more suitable for accurate speech recognition.

[0342] Step 2:

[0343] The terminal sends pre-processed audio data to the server. The server uses speech recognition technology to convert the audio data into text data. Here, the audio data is transformed into text format. This text data is then sent to the next stage.

[0344] Step 3:

[0345] The server feeds text data into a generative AI model, which then translates it into the specified target language. The translation is performed using prompts, and the text in the target language is output.

[0346] Step 4:

[0347] The text data translated on the server is then fed back into speech synthesis technology and converted back into speech data. Natural-sounding speech is generated using speech synthesis services such as Amazon Polly.

[0348] Step 5:

[0349] The generated audio data is sent from the server to the terminal. The terminal plays this audio data and provides it to the user. This allows the user to obtain audio information in their specified language.

[0350] Step 6:

[0351] When a user uses an image, the device sends the captured image data to the server. The server uses Tesseract OCR to extract text from the image and obtains this text data.

[0352] Step 7:

[0353] The server feeds the extracted text into a generative AI model for translation into the target language. The translated text is then processed to be placed on top of the original image.

[0354] Step 8:

[0355] Using OpenCV, the translation result is overlaid onto the original image. The device provides the user with the processed image data, allowing them to visually understand the information.

[0356] The above is a detailed processing flow of the system that implements the application example.

[0357] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0358] As an embodiment of the present invention, an example of providing a highly accurate real-time translation system that takes user emotions into consideration will be described. This system processes voice, text, and image input, recognizes the user's emotions using an emotion engine, and reflects the results in the translation.

[0359] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and normalization. Next, speech recognition technology is used to convert the voice into text, and this text information is passed to the generating AI and emotion engine. The emotion engine estimates the user's emotions from the tone of voice and nuances of speech. Using this emotion information, the server translates the text into the specified language using a translation model and makes corrections to appropriate wording and tone. The translation result is then converted into audio data through speech synthesis technology and sent to the device. The device plays this audio and provides it to the user.

[0360] In the case of text input, the user directly enters text into the device and sends it to the server. The server analyzes the sentiment of the text through its sentiment engine and passes the results to a translation model. Based on this sentiment information, it generates a translation result incorporating appropriate sentimental expressions and sends it back to the device. The device then displays the translated text on the screen and provides it to the user.

[0361] In image translation, the user uses their device to take or select an image and sends it to the server. The device uses OCR technology to extract text information from the image and sends this text to the server. The server uses an emotion engine to analyze the facial expressions of people and the context of the subjects in the image. This analysis is then incorporated into a translation model to produce an emotionally appropriate translation. The result is sent to the device and displayed as an overlay on the original image.

[0362] For example, if a user wants to translate an image sent by a friend, the emotion engine recognizes positive elements in the image, such as the smiles of the people in the image or the background. The server then uses this emotion information to perform the translation, generating and providing a more friendly and lighthearted translation. This allows the user to understand the context more deeply and achieve richer communication.

[0363] This system goes beyond simple language conversion, providing natural and accurate translations that reflect the user's emotions, thereby improving the quality of communication.

[0364] The following describes the processing flow.

[0365] Step 1:

[0366] The user inputs voice into the device, and the device captures that voice using its microphone. The device then performs preprocessing on this audio data, such as noise reduction and normalization.

[0367] Step 2:

[0368] The device uses speech recognition technology to convert pre-processed audio data into text data. This text data forms the basis for the translation process.

[0369] Step 3:

[0370] The terminal sends the converted text data to the server. At this time, the emotion engine extracts emotion data based on the tone and nuances unique to the voice.

[0371] Step 4:

[0372] The server uses an emotion engine to analyze emotional information obtained from the received text and determine the estimated emotional state of the user.

[0373] Step 5:

[0374] The server inputs text into a translation model, which then translates it into the specified language, taking into account the emotional information contained within. At this stage, the wording and tone are appropriately adjusted.

[0375] Step 6:

[0376] The server converts the translated text into speech data using speech synthesis technology, generating speech that reflects emotional expression.

[0377] Step 7:

[0378] The server sends the generated audio data to the terminal.

[0379] Step 8:

[0380] The device plays back the received audio data and provides the user with a translation result that reflects the emotions.

[0381] Step 9:

[0382] Users can view translated content through their devices and engage in natural communication.

[0383] (Example 2)

[0384] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0385] Traditional translation systems simply convert language, making it difficult to provide natural and accurate translations that take into account the user's emotions and context. Furthermore, there is a need to support different input formats such as audio, text, and images, while providing emotionally reflective translation results in real time.

[0386] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0387] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into text information using speech recognition technology; means for analyzing the emotions of the converted text information using emotion analysis technology; and means for converting the text information into a specified language using a translation model, taking the emotional information into consideration, and modifying the wording and tone to match the emotions. This makes it possible to provide translations that reflect the user's emotions in real time using different input formats.

[0388] "Voice input" is a method by which users provide information to a system through speaking.

[0389] "Preprocessing" refers to the initial processing performed to prepare input data for analysis and processing.

[0390] "Speech recognition technology" is a technology that converts speech into text data that can be understood by computers and other devices.

[0391] "Text information" refers to data represented as a string of characters, and is information used for language processing.

[0392] "Emotional analysis technology" is a technology that analyzes and extracts emotional states from text and audio data.

[0393] "Emotional information" refers to data that indicates the user's emotional state and is information that influences translation and other processing.

[0394] A "translation model" is an algorithm used to convert text from one language to another.

[0395] "Language choice" refers to the selection and tone of language used in communication.

[0396] "Intonation" refers to elements that represent the phonetic intonation and atmosphere of spoken language and written text.

[0397] "Speech synthesis technology" is a technology that converts text data into speech data.

[0398] "Audio data" refers to audio information expressed in digital format.

[0399] A "user" is a person or group that uses this system.

[0400] "Real-time" refers to processing that is performed with minimal delay and in an immediate manner.

[0401] "Different input formats" is a concept that includes a variety of data forms such as audio, text, and images.

[0402] This invention provides a highly accurate real-time translation system that takes user emotions into consideration. This system processes different input formats, namely voice, text, and images, estimates the user's emotions using sentiment analysis technology, and reflects the results in the translation.

[0403] When a user provides voice input, they provide their voice through the device's microphone. The device captures this voice data and performs noise reduction and normalization using libraries such as "LibROSA". Next, speech recognition technology such as "Google Speech-to-Text" converts the voice information into text information. This converted text information is sent to a server, which analyzes the sentiment using sentiment analysis technology such as "IBM Watson Tone Analyzer". Having obtained the sentiment information, the server uses a translation model (e.g., DeepL API) to convert the text information into the specified language, making adjustments to the wording and tone according to the sentiment. Subsequently, speech synthesis technology (e.g., "Amazon Polly") converts the text information back into voice data and sends it back to the device. The device provides this voice data to the user through its speaker.

[0404] In the case of text input, the user enters the content they wish to convert into the text input field on their device. The entered text information is sent to the server, which analyzes the sentiment of the text using sentiment analysis technology. Taking the sentiment information into account, the server uses a translation model to convert the text into appropriate language and expression, and sends the result back to the device. The device then displays the translation result on the screen and provides it to the user.

[0405] In image translation, the user uses their device to take or select an image they want to translate and sends it to the server. The server extracts text information from the image using optical character recognition (OCR) technology such as "Tesseract OCR" and evaluates the sentiment of the text and visual elements within the image using sentiment analysis technology. The server then inputs this sentiment information into a translation model and converts it into the appropriate language. The converted text is added as an overlay to the original image, and the device displays it to the user.

[0406] For example, if a user wants to translate a picture of a smiling person sent by a friend, the emotion analysis technology recognizes the smile of the subject in the image. The server interprets this as a positive emotion and reflects a friendly and approachable tone in the translation result. An example of a prompt would be, "Analyze this text with the emotion engine and provide a translation that matches the emotion."

[0407] This system goes beyond simple language conversion, enabling natural and accurate translations that reflect emotions, thereby improving the quality of communication.

[0408] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0409] Step 1:

[0410] Users input data as voice, text, or images using the device. For voice input, the user speaks through the device's microphone, and the device captures the voice data. For text input, the user directly enters text into the device's input field. For image input, the user prepares the image data by selecting the device's camera or a saved image.

[0411] Step 2:

[0412] The terminal performs noise reduction and normalization on the captured audio data. If the input data is audio, it uses a library such as "LibROSA" to perform preprocessing to convert the audio to text. This removes noise and outputs audio data that is easier to analyze.

[0413] Step 3:

[0414] The device performs speech recognition processing and converts the speech data into text information using technologies such as "Google Speech-to-Text". This conversion outputs the speech as text data that can be processed by a computer.

[0415] Step 4:

[0416] The server receives text data sent from the terminal and analyzes the sentiment of the text using sentiment analysis technology. Tools such as "IBM Watson Tone Analyzer" are used to estimate sentiment from the tone and context of the text and obtain sentiment information. This sentiment information influences the translation process.

[0417] Step 5:

[0418] The server considers the results of sentiment analysis and uses a translation model to convert the text data into the specified language. Translation models such as "DeepL API" are used to correct the wording and tone to suit the emotion. As a result of this process, the text is output as a translation that reflects the emotion.

[0419] Step 6:

[0420] The server uses speech synthesis technology to convert translated text information into audio data. Using services like "Amazon Polly," it converts the translated text into audio data and sends it to the terminal. Audio output is obtained at this stage.

[0421] Step 7:

[0422] The device receives audio data sent from the server and provides it to the user through the speaker. For text input, the translated text is displayed on the screen. For images, the translated text is displayed as an overlay and provided to the user. This process allows the user to receive translation results that are based on natural emotions.

[0423] (Application Example 2)

[0424] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0425] Current translation systems simply convert words into another language, making it difficult to provide natural translations that reflect the user's emotions. This often leads to a loss of emotional nuance in communication, particularly in multilingual households and intercultural exchanges, resulting in a decline in the quality of conversation. Conventional technologies lack mechanisms to recognize and apply emotions to translation, thus creating a need for technologies that enable more emotionally rich communication.

[0426] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0427] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into a sequence of symbols using acoustic recognition technology; means for identifying emotions and passing the estimated emotion information to a generation AI model; and means for translating the converted sequence of symbols into a specified symbol system using a translation model and adjusting the wording and tone based on the emotion information. This enables natural and accurate translation that reflects the user's emotions.

[0428] "Voice input" refers to the acoustic signals obtained when a user speaks, and it is a fundamental means for a system to receive information.

[0429] "Preprocessing" is the process of removing noise and normalizing audio and text data, and is preparatory work to improve recognition accuracy.

[0430] "Acoustic recognition technology" is a technology that analyzes audio data and generates text information from it; it is a process of converting human language into a format that computers can understand.

[0431] A "symbol sequence" is a sequence of characters or symbols that results from the processing and conversion of speech or text; it is an abstract form of information representation.

[0432] "Emotional identification" is a technology that identifies the emotional nuances behind input voice or text, and is a process for estimating the user's emotional state.

[0433] A "generative AI model" is a system that utilizes artificial intelligence to produce creative responses and outputs based on input data, and is a model designed to provide information that is appropriate to emotions and context.

[0434] A "translation model" is an algorithm or mechanism for converting text from one language to another, and it is a technology that acts as a bridge for conveying meaning between different languages.

[0435] A "designated symbolic system" refers to the target language or form of expression, and is a set of languages ​​and symbols defined in advance as the target of translation or processing.

[0436] "Speech synthesis technology" is a technology that artificially generates acoustic signals based on text data and outputs them as natural-sounding speech. It is a process that converts textual information into a form that humans can recognize by hearing.

[0437] "Audio data" refers to the digital format of sound generated by speech synthesis technology, and is the final output information provided to the user.

[0438] A "user" is a person or organization that receives services provided by the system, and is the entity that receives translation and information through interaction with the system.

[0439] As an embodiment of this invention, the system is constructed as follows. This system processes voice, text, and image input and provides real-time translation that takes emotion into account.

[0440] First, the user inputs voice into the device. The device captures the voice using its microphone and performs preprocessing such as noise reduction and normalization. Next, the device uses speech recognition technology (for example, the Google Speech-to-Text API) to convert the voice into text and sends that data to the server.

[0441] The server receives text data and uses an emotion analysis engine (for example, IBM Watson Tone Analyzer) to recognize emotions. This engine estimates the user's emotions from the tone of voice and nuances of speech, and passes that information to a generative AI model.

[0442] The generative AI model receives text to be translated based on sentiment information and uses a translation model (e.g., Google Translate API) to perform a translation that reflects natural and appropriate sentiment in the specified language. The translation result is then converted back into audio data using speech synthesis technology (e.g., Amazon Polly).

[0443] Finally, the device plays this audio data through its speaker and provides it to the user. This allows the user to receive the translated result, which reflects emotions, as audio.

[0444] As a concrete example, when a home robot conveys the gentle words of a parent who speaks Japanese to a child learning English, the voice assistant can analyze the parent's tone using an emotion engine, generate English with a positive nuance using an AI model, and provide a translation result that gives the child a sense of security.

[0445] As an example of a prompt, by inputting a request such as, "When Mom asks the robot, 'How was your homework today?', please convey the message in a gentle tone," the system can generate output corresponding to that instruction.

[0446] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0447] Step 1:

[0448] The user inputs voice into the device's microphone. The device captures the voice and performs initial preprocessing. This preprocessing applies a noise reduction filter and normalizes the audio signal, preparing it for high-quality data to be sent to the server. The input is raw audio data, and the output is normalized audio data with noise removed.

[0449] Step 2:

[0450] The device converts speech to text by sending pre-processed audio data to a speech recognition engine (e.g., Google Speech-to-Text API). The speech recognition engine analyzes the acoustic characteristics and generates corresponding string data. The input is de-noised audio data, and the output is the converted text data.

[0451] Step 3:

[0452] The terminal transfers the generated text data to the server. The server receives the text data and analyzes the sentiment information using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer). The sentiment analysis engine estimates the user's emotions from the structure and word choice of the text and outputs it as an emotion vector. The input is text data, and the output is an emotion vector.

[0453] Step 4:

[0454] The server sends emotion vectors to a generating AI model, which then passes them along with prompt text to a translation model (e.g., Google Translate API). This model considers the emotion information and translates the text into the specified language while maintaining appropriate tone and wording. The input is text data and emotion vectors, and the output is the translated text.

[0455] Step 5:

[0456] The server passes the translated text to a speech synthesis technology (e.g., Amazon Polly) to generate audio data. The speech synthesis engine synthesizes speech with natural tones and intonations that reflect emotional information, and outputs it as digital audio data. The input is the translated text, and the output is the synthesized audio data.

[0457] Step 6:

[0458] The terminal provides the user with audio data received from the server by playing it through its speaker. By listening to this, the user can receive a translation that reflects emotions. The input is synthesized audio data, and the output is the played audio.

[0459] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0460] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0461] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0462] [Third Embodiment]

[0463] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0464] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0465] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0466] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0467] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0468] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0469] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0470] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0471] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0472] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0473] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0474] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0475] As an embodiment of the present invention, an example is described in which an integrated platform for processing voice, text, and image inputs is established in a system that provides real-time, high-precision translation and interpretation. This system centrally processes various input methods and utilizes a generative AI model to provide multilingual translations.

[0476] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and voice normalization. Next, it converts the voice into text using speech recognition technology and sends it to the server. The server translates the received text into the specified language using a translation model. Then, it sends the translation result back to the device as audio data using speech synthesis technology. The device plays this audio data and provides it to the user.

[0477] In the case of text input, the user enters text into the terminal and sends it to the server. The server translates the entered text data into the specified language using a translation model. The translated result is sent back to the terminal and displayed on the terminal. The user can check the translation result on the screen.

[0478] In image translation, the user takes or selects an image using their device and sends it to the server. The device first converts the image data into text information using OCR technology. The server translates the extracted text into the specified language using a translation model and sends the result to the device. The device overlays the translated result onto the original image, allowing the user to visually confirm the text before and after translation.

[0479] For example, if a user wants to take a picture of a French menu and translate it into English, the image captured by the device is converted into text information using OCR. The server translates this text into English, and the device overlays the translation result onto the menu image for the user to see. The user can then understand the menu content based on this visual information.

[0480] This integrated system eliminates the complexities of traditional methods, allowing users to seamlessly process various inputs such as voice, text, and images, and obtain translation results quickly. This invention serves as a solution for removing language barriers in a variety of scenarios.

[0481] The following describes the processing flow.

[0482] Step 1:

[0483] The user inputs voice into the device, and the device captures that voice using its microphone. The device preprocesses the voice data through noise reduction and normalization.

[0484] Step 2:

[0485] The device converts pre-processed audio data into text data using speech recognition technology. This conversion allows it to understand spoken language as written text.

[0486] Step 3:

[0487] The terminal sends the converted text data to the server. A secure data transmission protocol is used to send the data.

[0488] Step 4:

[0489] The server inputs the received text data into a translation model and translates it into the specified language. This step ensures an appropriate translation that takes context and nuance into account.

[0490] Step 5:

[0491] The server converts the translation results into audio data using text-to-speech (TTS) technology. This process generates the translation results as natural-sounding speech.

[0492] Step 6:

[0493] The server sends the generated audio data to the terminal.

[0494] Step 7:

[0495] The device plays the received audio data through its speaker, conveying the translated content to the user.

[0496] Step 8:

[0497] Users can view and use the translation results through their devices.

[0498] (Example 1)

[0499] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0500] Translation systems that handle diverse input formats require individual process management for each format, leading to increased complexity and processing time for users. Furthermore, there is a demand for improved translation accuracy between different languages ​​and real-time processing.

[0501] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0502] In this invention, the server includes means for acquiring audio data, pre-processing it with data processing technology, and converting it into text data using speech recognition technology; means for translating the text data into a target language using a generative AI model; and means for converting the translated text data into a digital audio format and outputting it as audio using presentation technology. This enables centralized processing of various input formats and allows for highly accurate and real-time multilingual translation.

[0503] "Audio data" refers to sound information that is electronically captured, such as a user's speech, and represented in a digital format.

[0504] "Data processing technology" refers to operations such as filtering and noise reduction to adjust raw input data into a usable state.

[0505] "Speech recognition technology" is a technology for converting speech data into text format, and includes processing such as phonological analysis and language models.

[0506] "Text data" refers to character information expressed in a digital format, and is used for purposes such as translation and storage.

[0507] A "generative AI model" is an artificial intelligence system trained on a large dataset, capable of multilingual translation and creative text generation.

[0508] The "target language" is the language that is required as the translation result, to which the text data is converted during the translation process.

[0509] A "digital audio format" refers to an electronically represented format that converts text data into speech, and is audio data that can be played back on speakers or other devices.

[0510] "Presentation technology" refers to interfaces and media used to transmit converted information to users, and is a method of providing information through senses such as sight and hearing.

[0511] An "integrated platform" is a foundational system that centrally manages different input formats and processing technologies to provide an efficient and seamless user experience.

[0512] This invention relates to a system that integrates and processes multiple input formats to provide rapid multilingual translation. This system supports voice input, text input, and image input, and provides users with highly accurate translation results through processes appropriate to each format.

[0513] If the user selects voice input, the device captures voice data using its built-in microphone. The device then processes the voice data, performing noise reduction and normalization, and converts it into text data using speech recognition technology. A common speech recognition library is used for this process. The converted text data is sent to a server, which translates it into the target language using a generative AI model. For example, if the user inputs "Bonjour" by voice, the system will respond with "Hello" by voice.

[0514] In the case of text input, the user directly enters text into the device and sends it to the server. The server uses a generative AI model to translate the input text and sends the translated text back to the device. The user can then view the translated text on the screen.

[0515] For image input, the user either takes a picture with the device's camera or selects an existing image. The device uses OCR technology to extract text information from the image and generates text data. This data is sent to a server and translated by a generative AI model. The device then overlays the translated text onto the original image, allowing the user to visually understand the image. For example, in a scenario where a user takes a picture of a French sign and translates it into English, the word "Café" extracted from the image taken by the user will be displayed as "Cafe".

[0516] An example of a prompt would be, "Translate the French text in this image into English." This clearly communicates the translation request to the generative AI model and provides instructions for performing an appropriate translation.

[0517] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0518] Step 1:

[0519] Users can input information into the device in the form of voice, text, or images. For voice input, the user speaks into the microphone, and the device captures this as audio data. For text input, the user uses the device's keyboard to type. For image input, the user either takes a photo with the camera or selects an existing image.

[0520] Step 2:

[0521] The device applies a noise reduction filter to the captured audio data. This data processing removes external noise. Normalization adjusts the volume to maintain a constant amplitude in the audio signal. The input is raw audio data, and the output is de-noised audio data.

[0522] Step 3:

[0523] The device converts pre-processed audio data into text data using speech recognition software. For example, it converts the user's utterance "Hello, how are you?" into the text "Hello, how are you?". The input is devoiced audio data, and the output is the corresponding text.

[0524] Step 4:

[0525] When using image input, the device uses OCR technology to recognize characters within the image and extract text information. For example, from an image containing the word "Café," the text "Café" will be extracted. The input is image data, and the output is the text information within the image.

[0526] Step 5:

[0527] Text data (including data extracted by speech recognition or OCR) is sent from the terminal to the server. The input is the text data generated by the terminal, and the output is the data sent to the server.

[0528] Step 6:

[0529] The server translates the received text data using a generative AI model. The generative AI model is trained on a large language dataset and, as an example, translates "Bonjour" to "Hello". The input is the text data entered by the user, and the output is the translated text data.

[0530] Step 7:

[0531] The server sends the translation result to the terminal. The input is the translated text data, and the output is the translated data sent to the terminal.

[0532] Step 8:

[0533] The device uses speech synthesis software to convert translated text data into audio data and play it through the speaker. For example, "Hello" will be played in English. In the case of text input, the device displays the translation result on the screen. In the case of image input, the translation result is overlaid on the original image and presented to the user. The input is translated text data, and the output is either audio data or displayed data.

[0534] (Application Example 1)

[0535] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0536] Enabling smooth communication between different languages ​​is a crucial challenge in today's increasingly globalized society. Language barriers can be a major obstacle, especially during travel and cross-cultural exchange. Traditional translation methods require separate processing for audio, text, and image inputs, making them cumbersome for users. Therefore, there is a need for a new system that integrates these inputs and provides real-time translation results.

[0537] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0538] In this invention, the server includes means for receiving voice input and converting it into text using speech recognition technology, means for translating the text into a specified language using a translation model and converting it into voice data using speech synthesis technology, and means for processing image data, extracting character information using optical character recognition technology, and displaying the translated results on the image. This enables integrated processing of voice, text, and image information, and facilitates smooth communication between multiple languages.

[0539] "Voice input" is the process used to capture voice information into a digital device.

[0540] "Preprocessing" is the process of removing noise and unwanted parts from audio or text data so that subsequent processing can be performed more effectively.

[0541] "Speech recognition technology" is a technology that analyzes speech data and converts it into text information.

[0542] "Text" refers to a form of data expressed as character information.

[0543] A "translation model" is an algorithm or program used to convert input language data into another specified language.

[0544] "Speech synthesis technology" is a technology that converts text data into speech data and reproduces it in a way that sounds like human speech.

[0545] "Audio data" refers to sound information expressed in digital format.

[0546] A "user" is an individual or group that uses a system or service.

[0547] "Image data" refers to data that represents visual information in a digital format.

[0548] "Optical character recognition technology" is a technology that identifies character information from images and converts it into digital character data.

[0549] "Character information" refers to data content that consists of characters as its constituent elements.

[0550] "Display" refers to the process or means of presenting information visually.

[0551] "Multilingual travel support" refers to functions or services that enable travelers to understand and utilize information, overcoming language barriers.

[0552] The system that realizes this invention integrates speech recognition technology, optical character recognition technology, translation models, and speech synthesis technology to provide smooth multilingual translation.

[0553] The server receives audio, text, or image data over the network. For audio data, the server uses speech recognition technology to convert the speech to text. This process utilizes libraries such as Librosa and PyDub to denoise and normalize the audio. Then, a generative AI model is used to translate the text into the specified language. The translated text is then converted back into audio data using a speech synthesis service like Amazon Polly and sent to the user's device.

[0554] For image data, the server first extracts text information from the image using Tesseract OCR technology. The extracted text is then translated into the target language using a generative AI model. The translation result is overlaid on the original image using the OpenCV library and presented to the user visually.

[0555] Users can obtain multilingual travel assistance using smartphones and robots. For example, if a user wants to translate a French menu into English, an image taken with the device is converted into text information using OCR technology and then translated by a generative AI model. An example of a prompt would be, "Please use voice input to translate the French menu into English."

[0556] In this way, the system centrally processes diverse information inputs such as voice, text, and images, providing users with the ability to obtain information beyond language barriers.

[0557] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0558] Step 1:

[0559] The device accepts voice input from the user. After acquiring the voice data, Librosa is used to perform noise reduction and normalization of the speech. This preprocessing results in data that is more suitable for accurate speech recognition.

[0560] Step 2:

[0561] The terminal sends pre-processed audio data to the server. The server uses speech recognition technology to convert the audio data into text data. Here, the audio data is transformed into text format. This text data is then sent to the next stage.

[0562] Step 3:

[0563] The server feeds text data into a generative AI model, which then translates it into the specified target language. The translation is performed using prompts, and the text in the target language is output.

[0564] Step 4:

[0565] The text data translated on the server is then fed back into speech synthesis technology and converted back into speech data. Natural-sounding speech is generated using speech synthesis services such as Amazon Polly.

[0566] Step 5:

[0567] The generated audio data is sent from the server to the terminal. The terminal plays this audio data and provides it to the user. This allows the user to obtain audio information in their specified language.

[0568] Step 6:

[0569] When a user uses an image, the device sends the captured image data to the server. The server uses Tesseract OCR to extract text from the image and obtains this text data.

[0570] Step 7:

[0571] The server feeds the extracted text into a generative AI model for translation into the target language. The translated text is then processed to be placed on top of the original image.

[0572] Step 8:

[0573] Using OpenCV, the translation result is overlaid onto the original image. The device provides the user with the processed image data, allowing them to visually understand the information.

[0574] The above is a detailed processing flow of the system that implements the application example.

[0575] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0576] As an embodiment of the present invention, an example of providing a highly accurate real-time translation system that takes user emotions into consideration will be described. This system processes voice, text, and image input, recognizes the user's emotions using an emotion engine, and reflects the results in the translation.

[0577] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and normalization. Next, speech recognition technology is used to convert the voice into text, and this text information is passed to the generating AI and emotion engine. The emotion engine estimates the user's emotions from the tone of voice and nuances of speech. Using this emotion information, the server translates the text into the specified language using a translation model and makes corrections to appropriate wording and tone. The translation result is then converted into audio data through speech synthesis technology and sent to the device. The device plays this audio and provides it to the user.

[0578] In the case of text input, the user directly enters text into the device and sends it to the server. The server analyzes the sentiment of the text through its sentiment engine and passes the results to a translation model. Based on this sentiment information, it generates a translation result incorporating appropriate sentimental expressions and sends it back to the device. The device then displays the translated text on the screen and provides it to the user.

[0579] In image translation, the user uses their device to take or select an image and sends it to the server. The device uses OCR technology to extract text information from the image and sends this text to the server. The server uses an emotion engine to analyze the facial expressions of people and the context of the subjects in the image. This analysis is then incorporated into a translation model to produce an emotionally appropriate translation. The result is sent to the device and displayed as an overlay on the original image.

[0580] For example, if a user wants to translate an image sent by a friend, the emotion engine recognizes positive elements in the image, such as the smiles of the people in the image or the background. The server then uses this emotion information to perform the translation, generating and providing a more friendly and lighthearted translation. This allows the user to understand the context more deeply and achieve richer communication.

[0581] This system goes beyond simple language conversion, providing natural and accurate translations that reflect the user's emotions, thereby improving the quality of communication.

[0582] The following describes the processing flow.

[0583] Step 1:

[0584] The user inputs voice into the device, and the device captures that voice using its microphone. The device then performs preprocessing on this audio data, such as noise reduction and normalization.

[0585] Step 2:

[0586] The device uses speech recognition technology to convert pre-processed audio data into text data. This text data forms the basis for the translation process.

[0587] Step 3:

[0588] The terminal sends the converted text data to the server. At this time, the emotion engine extracts emotion data based on the tone and nuances unique to the voice.

[0589] Step 4:

[0590] The server uses an emotion engine to analyze emotional information obtained from the received text and determine the estimated emotional state of the user.

[0591] Step 5:

[0592] The server inputs text into a translation model, which then translates it into the specified language, taking into account the emotional information contained within. At this stage, the wording and tone are appropriately adjusted.

[0593] Step 6:

[0594] The server converts the translated text into speech data using speech synthesis technology, generating speech that reflects emotional expression.

[0595] Step 7:

[0596] The server sends the generated audio data to the terminal.

[0597] Step 8:

[0598] The device plays back the received audio data and provides the user with a translation result that reflects the emotions.

[0599] Step 9:

[0600] Users can view translated content through their devices and engage in natural communication.

[0601] (Example 2)

[0602] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0603] Traditional translation systems simply convert language, making it difficult to provide natural and accurate translations that take into account the user's emotions and context. Furthermore, there is a need to support different input formats such as audio, text, and images, while providing emotionally reflective translation results in real time.

[0604] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0605] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into text information using speech recognition technology; means for analyzing the emotions of the converted text information using emotion analysis technology; and means for converting the text information into a specified language using a translation model, taking the emotional information into consideration, and modifying the wording and tone to match the emotions. This makes it possible to provide translations that reflect the user's emotions in real time using different input formats.

[0606] "Voice input" is a method by which users provide information to a system through speaking.

[0607] "Preprocessing" refers to the initial processing performed to prepare input data for analysis and processing.

[0608] "Speech recognition technology" is a technology that converts speech into text data that can be understood by computers and other devices.

[0609] "Text information" refers to data represented as a string of characters, and is information used for language processing.

[0610] "Emotional analysis technology" is a technology that analyzes and extracts emotional states from text and audio data.

[0611] "Emotional information" refers to data that indicates the user's emotional state and is information that influences translation and other processing.

[0612] A "translation model" is an algorithm used to convert text from one language to another.

[0613] "Language choice" refers to the selection and tone of language used in communication.

[0614] "Intonation" refers to elements that represent the phonetic intonation and atmosphere of spoken language and written text.

[0615] "Speech synthesis technology" is a technology that converts text data into speech data.

[0616] "Audio data" refers to audio information expressed in digital format.

[0617] A "user" is a person or group that uses this system.

[0618] "Real-time" refers to processing that is performed with minimal delay and in an immediate manner.

[0619] "Different input formats" is a concept that includes a variety of data forms such as audio, text, and images.

[0620] This invention provides a highly accurate real-time translation system that takes user emotions into consideration. This system processes different input formats, namely voice, text, and images, estimates the user's emotions using sentiment analysis technology, and reflects the results in the translation.

[0621] When a user provides voice input, they provide their voice through the device's microphone. The device captures this voice data and performs noise reduction and normalization using libraries such as "LibROSA". Next, speech recognition technology such as "Google Speech-to-Text" converts the voice information into text information. This converted text information is sent to a server, which analyzes the sentiment using sentiment analysis technology such as "IBM Watson Tone Analyzer". Having obtained the sentiment information, the server uses a translation model (e.g., DeepL API) to convert the text information into the specified language, making adjustments to the wording and tone according to the sentiment. Subsequently, speech synthesis technology (e.g., "Amazon Polly") converts the text information back into voice data and sends it back to the device. The device provides this voice data to the user through its speaker.

[0622] In the case of text input, the user enters the content they wish to convert into the text input field on their device. The entered text information is sent to the server, which analyzes the sentiment of the text using sentiment analysis technology. Taking the sentiment information into account, the server uses a translation model to convert the text into appropriate language and expression, and sends the result back to the device. The device then displays the translation result on the screen and provides it to the user.

[0623] In image translation, the user uses their device to take or select an image they want to translate and sends it to the server. The server extracts text information from the image using optical character recognition (OCR) technology such as "Tesseract OCR" and evaluates the sentiment of the text and visual elements within the image using sentiment analysis technology. The server then inputs this sentiment information into a translation model and converts it into the appropriate language. The converted text is added as an overlay to the original image, and the device displays it to the user.

[0624] For example, if a user wants to translate a picture of a smiling person sent by a friend, the emotion analysis technology recognizes the smile of the subject in the image. The server interprets this as a positive emotion and reflects a friendly and approachable tone in the translation result. An example of a prompt would be, "Analyze this text with the emotion engine and provide a translation that matches the emotion."

[0625] This system goes beyond simple language conversion, enabling natural and accurate translations that reflect emotions, thereby improving the quality of communication.

[0626] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0627] Step 1:

[0628] Users input data as voice, text, or images using the device. For voice input, the user speaks through the device's microphone, and the device captures the voice data. For text input, the user directly enters text into the device's input field. For image input, the user prepares the image data by selecting the device's camera or a saved image.

[0629] Step 2:

[0630] The terminal performs noise reduction and normalization on the captured audio data. If the input data is audio, it uses a library such as "LibROSA" to perform preprocessing to convert the audio to text. This removes noise and outputs audio data that is easier to analyze.

[0631] Step 3:

[0632] The device performs speech recognition processing and converts the speech data into text information using technologies such as "Google Speech-to-Text". This conversion outputs the speech as text data that can be processed by a computer.

[0633] Step 4:

[0634] The server receives text data sent from the terminal and analyzes the sentiment of the text using sentiment analysis technology. Tools such as "IBM Watson Tone Analyzer" are used to estimate sentiment from the tone and context of the text and obtain sentiment information. This sentiment information influences the translation process.

[0635] Step 5:

[0636] The server considers the results of sentiment analysis and uses a translation model to convert the text data into the specified language. Translation models such as "DeepL API" are used to correct the wording and tone to suit the emotion. As a result of this process, the text is output as a translation that reflects the emotion.

[0637] Step 6:

[0638] The server uses speech synthesis technology to convert translated text information into audio data. Using services like "Amazon Polly," it converts the translated text into audio data and sends it to the terminal. Audio output is obtained at this stage.

[0639] Step 7:

[0640] The device receives audio data sent from the server and provides it to the user through the speaker. For text input, the translated text is displayed on the screen. For images, the translated text is displayed as an overlay and provided to the user. This process allows the user to receive translation results that are based on natural emotions.

[0641] (Application Example 2)

[0642] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0643] Current translation systems simply convert words into another language, making it difficult to provide natural translations that reflect the user's emotions. This often leads to a loss of emotional nuance in communication, particularly in multilingual households and intercultural exchanges, resulting in a decline in the quality of conversation. Conventional technologies lack mechanisms to recognize and apply emotions to translation, thus creating a need for technologies that enable more emotionally rich communication.

[0644] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0645] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into a sequence of symbols using acoustic recognition technology; means for identifying emotions and passing the estimated emotion information to a generation AI model; and means for translating the converted sequence of symbols into a specified symbol system using a translation model and adjusting the wording and tone based on the emotion information. This enables natural and accurate translation that reflects the user's emotions.

[0646] "Voice input" refers to the acoustic signals obtained when a user speaks, and it is a fundamental means for a system to receive information.

[0647] "Preprocessing" is the process of removing noise and normalizing audio and text data, and is preparatory work to improve recognition accuracy.

[0648] "Acoustic recognition technology" is a technology that analyzes audio data and generates text information from it; it is a process of converting human language into a format that computers can understand.

[0649] A "symbol sequence" is a sequence of characters or symbols that results from the processing and conversion of speech or text; it is an abstract form of information representation.

[0650] "Emotional identification" is a technology that identifies the emotional nuances behind input voice or text, and is a process for estimating the user's emotional state.

[0651] A "generative AI model" is a system that utilizes artificial intelligence to produce creative responses and outputs based on input data, and is a model designed to provide information that is appropriate to emotions and context.

[0652] A "translation model" is an algorithm or mechanism for converting text from one language to another, and it is a technology that acts as a bridge for conveying meaning between different languages.

[0653] A "designated symbolic system" refers to the target language or form of expression, and is a set of languages ​​and symbols defined in advance as the target of translation or processing.

[0654] "Speech synthesis technology" is a technology that artificially generates acoustic signals based on text data and outputs them as natural-sounding speech. It is a process that converts textual information into a form that humans can recognize by hearing.

[0655] "Audio data" refers to the digital format of sound generated by speech synthesis technology, and is the final output information provided to the user.

[0656] A "user" is a person or organization that receives services provided by the system, and is the entity that receives translation and information through interaction with the system.

[0657] As an embodiment of this invention, the system is constructed as follows. This system processes voice, text, and image input and provides real-time translation that takes emotion into account.

[0658] First, the user inputs voice into the device. The device captures the voice using its microphone and performs preprocessing such as noise reduction and normalization. Next, the device uses speech recognition technology (for example, the Google Speech-to-Text API) to convert the voice into text and sends that data to the server.

[0659] The server receives text data and uses an emotion analysis engine (for example, IBM Watson Tone Analyzer) to recognize emotions. This engine estimates the user's emotions from the tone of voice and nuances of speech, and passes that information to a generative AI model.

[0660] The generative AI model receives text to be translated based on sentiment information and uses a translation model (e.g., Google Translate API) to perform a translation that reflects natural and appropriate sentiment in the specified language. The translation result is then converted back into audio data using speech synthesis technology (e.g., Amazon Polly).

[0661] Finally, the device plays this audio data through its speaker and provides it to the user. This allows the user to receive the translated result, which reflects emotions, as audio.

[0662] As a concrete example, when a home robot conveys the gentle words of a parent who speaks Japanese to a child learning English, the voice assistant can analyze the parent's tone using an emotion engine, generate English with a positive nuance using an AI model, and provide a translation result that gives the child a sense of security.

[0663] As an example of a prompt, by inputting a request such as, "When Mom asks the robot, 'How was your homework today?', please convey the message in a gentle tone," the system can generate output corresponding to that instruction.

[0664] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0665] Step 1:

[0666] The user inputs voice into the device's microphone. The device captures the voice and performs initial preprocessing. This preprocessing applies a noise reduction filter and normalizes the audio signal, preparing it for high-quality data to be sent to the server. The input is raw audio data, and the output is normalized audio data with noise removed.

[0667] Step 2:

[0668] The device converts speech to text by sending pre-processed audio data to a speech recognition engine (e.g., Google Speech-to-Text API). The speech recognition engine analyzes the acoustic characteristics and generates corresponding string data. The input is de-noised audio data, and the output is the converted text data.

[0669] Step 3:

[0670] The terminal transfers the generated text data to the server. The server receives the text data and analyzes the sentiment information using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer). The sentiment analysis engine estimates the user's emotions from the structure and word choice of the text and outputs it as an emotion vector. The input is text data, and the output is an emotion vector.

[0671] Step 4:

[0672] The server sends emotion vectors to a generating AI model, which then passes them along with prompt text to a translation model (e.g., Google Translate API). This model considers the emotion information and translates the text into the specified language while maintaining appropriate tone and wording. The input is text data and emotion vectors, and the output is the translated text.

[0673] Step 5:

[0674] The server passes the translated text to a speech synthesis technology (e.g., Amazon Polly) to generate audio data. The speech synthesis engine synthesizes speech with natural tones and intonations that reflect emotional information, and outputs it as digital audio data. The input is the translated text, and the output is the synthesized audio data.

[0675] Step 6:

[0676] The terminal provides the user with audio data received from the server by playing it through its speaker. By listening to this, the user can receive a translation that reflects emotions. The input is synthesized audio data, and the output is the played audio.

[0677] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0678] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0679] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0680] [Fourth Embodiment]

[0681] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0682] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0683] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0684] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0685] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0686] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0687] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0688] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0689] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0690] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0691] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0692] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0693] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0694] As an embodiment of the present invention, an example is described in which an integrated platform for processing voice, text, and image inputs is established in a system that provides real-time, high-precision translation and interpretation. This system centrally processes various input methods and utilizes a generative AI model to provide multilingual translations.

[0695] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and voice normalization. Next, it converts the voice into text using speech recognition technology and sends it to the server. The server translates the received text into the specified language using a translation model. Then, it sends the translation result back to the device as audio data using speech synthesis technology. The device plays this audio data and provides it to the user.

[0696] In the case of text input, the user enters text into the terminal and sends it to the server. The server translates the entered text data into the specified language using a translation model. The translated result is sent back to the terminal and displayed on the terminal. The user can check the translation result on the screen.

[0697] In image translation, the user takes or selects an image using their device and sends it to the server. The device first converts the image data into text information using OCR technology. The server translates the extracted text into the specified language using a translation model and sends the result to the device. The device overlays the translated result onto the original image, allowing the user to visually confirm the text before and after translation.

[0698] For example, if a user wants to take a picture of a French menu and translate it into English, the image captured by the device is converted into text information using OCR. The server translates this text into English, and the device overlays the translation result onto the menu image for the user to see. The user can then understand the menu content based on this visual information.

[0699] This integrated system eliminates the complexities of traditional methods, allowing users to seamlessly process various inputs such as voice, text, and images, and obtain translation results quickly. This invention serves as a solution for removing language barriers in a variety of scenarios.

[0700] The following describes the processing flow.

[0701] Step 1:

[0702] The user inputs voice into the device, and the device captures that voice using its microphone. The device preprocesses the voice data through noise reduction and normalization.

[0703] Step 2:

[0704] The device converts pre-processed audio data into text data using speech recognition technology. This conversion allows it to understand spoken language as written text.

[0705] Step 3:

[0706] The terminal sends the converted text data to the server. A secure data transmission protocol is used to send the data.

[0707] Step 4:

[0708] The server inputs the received text data into a translation model and translates it into the specified language. This step ensures an appropriate translation that takes context and nuance into account.

[0709] Step 5:

[0710] The server converts the translation results into audio data using text-to-speech (TTS) technology. This process generates the translation results as natural-sounding speech.

[0711] Step 6:

[0712] The server sends the generated audio data to the terminal.

[0713] Step 7:

[0714] The device plays the received audio data through its speaker, conveying the translated content to the user.

[0715] Step 8:

[0716] Users can view and use the translation results through their devices.

[0717] (Example 1)

[0718] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0719] Translation systems that handle diverse input formats require individual process management for each format, leading to increased complexity and processing time for users. Furthermore, there is a demand for improved translation accuracy between different languages ​​and real-time processing.

[0720] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0721] In this invention, the server includes means for acquiring audio data, pre-processing it with data processing technology, and converting it into text data using speech recognition technology; means for translating the text data into a target language using a generative AI model; and means for converting the translated text data into a digital audio format and outputting it as audio using presentation technology. This enables centralized processing of various input formats and allows for highly accurate and real-time multilingual translation.

[0722] "Audio data" refers to sound information that is electronically captured, such as a user's speech, and represented in a digital format.

[0723] "Data processing technology" refers to operations such as filtering and noise reduction to adjust raw input data into a usable state.

[0724] "Speech recognition technology" is a technology for converting speech data into text format, and includes processing such as phonological analysis and language models.

[0725] "Text data" refers to character information expressed in a digital format, and is used for purposes such as translation and storage.

[0726] A "generative AI model" is an artificial intelligence system trained on a large dataset, capable of multilingual translation and creative text generation.

[0727] The "target language" is the language that is required as the translation result, to which the text data is converted during the translation process.

[0728] A "digital audio format" refers to an electronically represented format that converts text data into speech, and is audio data that can be played back on speakers or other devices.

[0729] "Presentation technology" refers to interfaces and media used to transmit converted information to users, and is a method of providing information through senses such as sight and hearing.

[0730] An "integrated platform" is a foundational system that centrally manages different input formats and processing technologies to provide an efficient and seamless user experience.

[0731] This invention relates to a system that integrates and processes multiple input formats to provide rapid multilingual translation. This system supports voice input, text input, and image input, and provides users with highly accurate translation results through processes appropriate to each format.

[0732] If the user selects voice input, the device captures voice data using its built-in microphone. The device then processes the voice data, performing noise reduction and normalization, and converts it into text data using speech recognition technology. A common speech recognition library is used for this process. The converted text data is sent to a server, which translates it into the target language using a generative AI model. For example, if the user inputs "Bonjour" by voice, the system will respond with "Hello" by voice.

[0733] In the case of text input, the user directly enters text into the device and sends it to the server. The server uses a generative AI model to translate the input text and sends the translated text back to the device. The user can then view the translated text on the screen.

[0734] For image input, the user either takes a picture with the device's camera or selects an existing image. The device uses OCR technology to extract text information from the image and generates text data. This data is sent to a server and translated by a generative AI model. The device then overlays the translated text onto the original image, allowing the user to visually understand the image. For example, in a scenario where a user takes a picture of a French sign and translates it into English, the word "Café" extracted from the image taken by the user will be displayed as "Cafe".

[0735] An example of a prompt would be, "Translate the French text in this image into English." This clearly communicates the translation request to the generative AI model and provides instructions for performing an appropriate translation.

[0736] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0737] Step 1:

[0738] Users can input information into the device in the form of voice, text, or images. For voice input, the user speaks into the microphone, and the device captures this as audio data. For text input, the user uses the device's keyboard to type. For image input, the user either takes a photo with the camera or selects an existing image.

[0739] Step 2:

[0740] The device applies a noise reduction filter to the captured audio data. This data processing removes external noise. Normalization adjusts the volume to maintain a constant amplitude in the audio signal. The input is raw audio data, and the output is de-noised audio data.

[0741] Step 3:

[0742] The device converts pre-processed audio data into text data using speech recognition software. For example, it converts the user's utterance "Hello, how are you?" into the text "Hello, how are you?". The input is devoiced audio data, and the output is the corresponding text.

[0743] Step 4:

[0744] When using image input, the device uses OCR technology to recognize characters within the image and extract text information. For example, from an image containing the word "Café," the text "Café" will be extracted. The input is image data, and the output is the text information within the image.

[0745] Step 5:

[0746] Text data (including data extracted by speech recognition or OCR) is sent from the terminal to the server. The input is the text data generated by the terminal, and the output is the data sent to the server.

[0747] Step 6:

[0748] The server translates the received text data using a generative AI model. The generative AI model is trained on a large language dataset and, as an example, translates "Bonjour" to "Hello". The input is the text data entered by the user, and the output is the translated text data.

[0749] Step 7:

[0750] The server sends the translation result to the terminal. The input is the translated text data, and the output is the translated data sent to the terminal.

[0751] Step 8:

[0752] The device uses speech synthesis software to convert translated text data into audio data and play it through the speaker. For example, "Hello" will be played in English. In the case of text input, the device displays the translation result on the screen. In the case of image input, the translation result is overlaid on the original image and presented to the user. The input is translated text data, and the output is either audio data or displayed data.

[0753] (Application Example 1)

[0754] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0755] Enabling smooth communication between different languages ​​is a crucial challenge in today's increasingly globalized society. Language barriers can be a major obstacle, especially during travel and cross-cultural exchange. Traditional translation methods require separate processing for audio, text, and image inputs, making them cumbersome for users. Therefore, there is a need for a new system that integrates these inputs and provides real-time translation results.

[0756] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0757] In this invention, the server includes means for receiving voice input and converting it into text using speech recognition technology, means for translating the text into a specified language using a translation model and converting it into voice data using speech synthesis technology, and means for processing image data, extracting character information using optical character recognition technology, and displaying the translated results on the image. This enables integrated processing of voice, text, and image information, and facilitates smooth communication between multiple languages.

[0758] "Voice input" is the process used to capture voice information into a digital device.

[0759] "Preprocessing" is the process of removing noise and unwanted parts from audio or text data so that subsequent processing can be performed more effectively.

[0760] "Speech recognition technology" is a technology that analyzes speech data and converts it into text information.

[0761] "Text" refers to a form of data expressed as character information.

[0762] A "translation model" is an algorithm or program used to convert input language data into another specified language.

[0763] "Speech synthesis technology" is a technology that converts text data into speech data and reproduces it in a way that sounds like human speech.

[0764] "Audio data" refers to sound information expressed in digital format.

[0765] A "user" is an individual or group that uses a system or service.

[0766] "Image data" refers to data that represents visual information in a digital format.

[0767] "Optical character recognition technology" is a technology that identifies character information from images and converts it into digital character data.

[0768] "Character information" refers to data content that consists of characters as its constituent elements.

[0769] "Display" refers to the process or means of presenting information visually.

[0770] "Multilingual travel support" refers to functions or services that enable travelers to understand and utilize information, overcoming language barriers.

[0771] The system that realizes this invention integrates speech recognition technology, optical character recognition technology, translation models, and speech synthesis technology to provide smooth multilingual translation.

[0772] The server receives audio, text, or image data over the network. For audio data, the server uses speech recognition technology to convert the speech to text. This process utilizes libraries such as Librosa and PyDub to denoise and normalize the audio. Then, a generative AI model is used to translate the text into the specified language. The translated text is then converted back into audio data using a speech synthesis service like Amazon Polly and sent to the user's device.

[0773] For image data, the server first extracts text information from the image using Tesseract OCR technology. The extracted text is then translated into the target language using a generative AI model. The translation result is overlaid on the original image using the OpenCV library and presented to the user visually.

[0774] Users can obtain multilingual travel assistance using smartphones and robots. For example, if a user wants to translate a French menu into English, an image taken with the device is converted into text information using OCR technology and then translated by a generative AI model. An example of a prompt would be, "Please use voice input to translate the French menu into English."

[0775] In this way, the system centrally processes diverse information inputs such as voice, text, and images, providing users with the ability to obtain information beyond language barriers.

[0776] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0777] Step 1:

[0778] The device accepts voice input from the user. After acquiring the voice data, Librosa is used to perform noise reduction and normalization of the speech. This preprocessing results in data that is more suitable for accurate speech recognition.

[0779] Step 2:

[0780] The terminal sends pre-processed audio data to the server. The server uses speech recognition technology to convert the audio data into text data. Here, the audio data is transformed into text format. This text data is then sent to the next stage.

[0781] Step 3:

[0782] The server feeds text data into a generative AI model, which then translates it into the specified target language. The translation is performed using prompts, and the text in the target language is output.

[0783] Step 4:

[0784] The text data translated on the server is then fed back into speech synthesis technology and converted back into speech data. Natural-sounding speech is generated using speech synthesis services such as Amazon Polly.

[0785] Step 5:

[0786] The generated audio data is sent from the server to the terminal. The terminal plays this audio data and provides it to the user. This allows the user to obtain audio information in their specified language.

[0787] Step 6:

[0788] When a user uses an image, the device sends the captured image data to the server. The server uses Tesseract OCR to extract text from the image and obtains this text data.

[0789] Step 7:

[0790] The server feeds the extracted text into a generative AI model for translation into the target language. The translated text is then processed to be placed on top of the original image.

[0791] Step 8:

[0792] Using OpenCV, the translation result is overlaid onto the original image. The device provides the user with the processed image data, allowing them to visually understand the information.

[0793] The above is a detailed processing flow of the system that implements the application example.

[0794] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0795] As an embodiment of the present invention, an example of providing a highly accurate real-time translation system that takes user emotions into consideration will be described. This system processes voice, text, and image input, recognizes the user's emotions using an emotion engine, and reflects the results in the translation.

[0796] First, let's explain the case where the user uses voice input. When the user speaks into the device, the device captures the voice and performs preprocessing. Preprocessing includes noise reduction and normalization. Next, speech recognition technology is used to convert the voice into text, and this text information is passed to the generating AI and emotion engine. The emotion engine estimates the user's emotions from the tone of voice and nuances of speech. Using this emotion information, the server translates the text into the specified language using a translation model and makes corrections to appropriate wording and tone. The translation result is then converted into audio data through speech synthesis technology and sent to the device. The device plays this audio and provides it to the user.

[0797] In the case of text input, the user directly enters text into the device and sends it to the server. The server analyzes the sentiment of the text through its sentiment engine and passes the results to a translation model. Based on this sentiment information, it generates a translation result incorporating appropriate sentimental expressions and sends it back to the device. The device then displays the translated text on the screen and provides it to the user.

[0798] In image translation, the user uses their device to take or select an image and sends it to the server. The device uses OCR technology to extract text information from the image and sends this text to the server. The server uses an emotion engine to analyze the facial expressions of people and the context of the subjects in the image. This analysis is then incorporated into a translation model to produce an emotionally appropriate translation. The result is sent to the device and displayed as an overlay on the original image.

[0799] For example, if a user wants to translate an image sent by a friend, the emotion engine recognizes positive elements in the image, such as the smiles of the people in the image or the background. The server then uses this emotion information to perform the translation, generating and providing a more friendly and lighthearted translation. This allows the user to understand the context more deeply and achieve richer communication.

[0800] This system goes beyond simple language conversion, providing natural and accurate translations that reflect the user's emotions, thereby improving the quality of communication.

[0801] The following describes the processing flow.

[0802] Step 1:

[0803] The user inputs voice into the device, and the device captures that voice using its microphone. The device then performs preprocessing on this audio data, such as noise reduction and normalization.

[0804] Step 2:

[0805] The device uses speech recognition technology to convert pre-processed audio data into text data. This text data forms the basis for the translation process.

[0806] Step 3:

[0807] The terminal sends the converted text data to the server. At this time, the emotion engine extracts emotion data based on the tone and nuances unique to the voice.

[0808] Step 4:

[0809] The server uses an emotion engine to analyze emotional information obtained from the received text and determine the estimated emotional state of the user.

[0810] Step 5:

[0811] The server inputs text into a translation model, which then translates it into the specified language, taking into account the emotional information contained within. At this stage, the wording and tone are appropriately adjusted.

[0812] Step 6:

[0813] The server converts the translated text into speech data using speech synthesis technology, generating speech that reflects emotional expression.

[0814] Step 7:

[0815] The server sends the generated audio data to the terminal.

[0816] Step 8:

[0817] The device plays back the received audio data and provides the user with a translation result that reflects the emotions.

[0818] Step 9:

[0819] Users can view translated content through their devices and engage in natural communication.

[0820] (Example 2)

[0821] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0822] Traditional translation systems simply convert language, making it difficult to provide natural and accurate translations that take into account the user's emotions and context. Furthermore, there is a need to support different input formats such as audio, text, and images, while providing emotionally reflective translation results in real time.

[0823] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0824] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into text information using speech recognition technology; means for analyzing the emotions of the converted text information using emotion analysis technology; and means for converting the text information into a specified language using a translation model, taking the emotional information into consideration, and modifying the wording and tone to match the emotions. This makes it possible to provide translations that reflect the user's emotions in real time using different input formats.

[0825] "Voice input" is a method by which users provide information to a system through speaking.

[0826] "Preprocessing" refers to the initial processing performed to prepare input data for analysis and processing.

[0827] "Speech recognition technology" is a technology that converts speech into text data that can be understood by computers and other devices.

[0828] "Text information" refers to data represented as a string of characters, and is information used for language processing.

[0829] "Emotional analysis technology" is a technology that analyzes and extracts emotional states from text and audio data.

[0830] "Emotional information" refers to data that indicates the user's emotional state and is information that influences translation and other processing.

[0831] A "translation model" is an algorithm used to convert text from one language to another.

[0832] "Language choice" refers to the selection and tone of language used in communication.

[0833] "Intonation" refers to elements that represent the phonetic intonation and atmosphere of spoken language and written text.

[0834] "Speech synthesis technology" is a technology that converts text data into speech data.

[0835] "Audio data" refers to audio information expressed in digital format.

[0836] A "user" is a person or group that uses this system.

[0837] "Real-time" refers to processing that is performed with minimal delay and in an immediate manner.

[0838] "Different input formats" is a concept that includes a variety of data forms such as audio, text, and images.

[0839] This invention provides a highly accurate real-time translation system that takes user emotions into consideration. This system processes different input formats, namely voice, text, and images, estimates the user's emotions using sentiment analysis technology, and reflects the results in the translation.

[0840] When a user provides voice input, they provide their voice through the device's microphone. The device captures this voice data and performs noise reduction and normalization using libraries such as "LibROSA". Next, speech recognition technology such as "Google Speech-to-Text" converts the voice information into text information. This converted text information is sent to a server, which analyzes the sentiment using sentiment analysis technology such as "IBM Watson Tone Analyzer". Having obtained the sentiment information, the server uses a translation model (e.g., DeepL API) to convert the text information into the specified language, making adjustments to the wording and tone according to the sentiment. Subsequently, speech synthesis technology (e.g., "Amazon Polly") converts the text information back into voice data and sends it back to the device. The device provides this voice data to the user through its speaker.

[0841] In the case of text input, the user enters the content they wish to convert into the text input field on their device. The entered text information is sent to the server, which analyzes the sentiment of the text using sentiment analysis technology. Taking the sentiment information into account, the server uses a translation model to convert the text into appropriate language and expression, and sends the result back to the device. The device then displays the translation result on the screen and provides it to the user.

[0842] In image translation, the user uses their device to take or select an image they want to translate and sends it to the server. The server extracts text information from the image using optical character recognition (OCR) technology such as "Tesseract OCR" and evaluates the sentiment of the text and visual elements within the image using sentiment analysis technology. The server then inputs this sentiment information into a translation model and converts it into the appropriate language. The converted text is added as an overlay to the original image, and the device displays it to the user.

[0843] For example, if a user wants to translate a picture of a smiling person sent by a friend, the emotion analysis technology recognizes the smile of the subject in the image. The server interprets this as a positive emotion and reflects a friendly and approachable tone in the translation result. An example of a prompt would be, "Analyze this text with the emotion engine and provide a translation that matches the emotion."

[0844] This system goes beyond simple language conversion, enabling natural and accurate translations that reflect emotions, thereby improving the quality of communication.

[0845] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0846] Step 1:

[0847] Users input data as voice, text, or images using the device. For voice input, the user speaks through the device's microphone, and the device captures the voice data. For text input, the user directly enters text into the device's input field. For image input, the user prepares the image data by selecting the device's camera or a saved image.

[0848] Step 2:

[0849] The terminal performs noise reduction and normalization on the captured audio data. If the input data is audio, it uses a library such as "LibROSA" to perform preprocessing to convert the audio to text. This removes noise and outputs audio data that is easier to analyze.

[0850] Step 3:

[0851] The device performs speech recognition processing and converts the speech data into text information using technologies such as "Google Speech-to-Text". This conversion outputs the speech as text data that can be processed by a computer.

[0852] Step 4:

[0853] The server receives text data sent from the terminal and analyzes the sentiment of the text using sentiment analysis technology. Tools such as "IBM Watson Tone Analyzer" are used to estimate sentiment from the tone and context of the text and obtain sentiment information. This sentiment information influences the translation process.

[0854] Step 5:

[0855] The server considers the results of sentiment analysis and uses a translation model to convert the text data into the specified language. Translation models such as "DeepL API" are used to correct the wording and tone to suit the emotion. As a result of this process, the text is output as a translation that reflects the emotion.

[0856] Step 6:

[0857] The server uses speech synthesis technology to convert translated text information into audio data. Using services like "Amazon Polly," it converts the translated text into audio data and sends it to the terminal. Audio output is obtained at this stage.

[0858] Step 7:

[0859] The device receives audio data sent from the server and provides it to the user through the speaker. For text input, the translated text is displayed on the screen. For images, the translated text is displayed as an overlay and provided to the user. This process allows the user to receive translation results that are based on natural emotions.

[0860] (Application Example 2)

[0861] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0862] Current translation systems simply convert words into another language, making it difficult to provide natural translations that reflect the user's emotions. This often leads to a loss of emotional nuance in communication, particularly in multilingual households and intercultural exchanges, resulting in a decline in the quality of conversation. Conventional technologies lack mechanisms to recognize and apply emotions to translation, thus creating a need for technologies that enable more emotionally rich communication.

[0863] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0864] In this invention, the server includes means for receiving voice input, performing preprocessing, and converting the voice into a sequence of symbols using acoustic recognition technology; means for identifying emotions and passing the estimated emotion information to a generation AI model; and means for translating the converted sequence of symbols into a specified symbol system using a translation model and adjusting the wording and tone based on the emotion information. This enables natural and accurate translation that reflects the user's emotions.

[0865] "Voice input" refers to the acoustic signals obtained when a user speaks, and it is a fundamental means for a system to receive information.

[0866] "Preprocessing" is the process of removing noise and normalizing audio and text data, and is preparatory work to improve recognition accuracy.

[0867] "Acoustic recognition technology" is a technology that analyzes audio data and generates text information from it; it is a process of converting human language into a format that computers can understand.

[0868] A "symbol sequence" is a sequence of characters or symbols that results from the processing and conversion of speech or text; it is an abstract form of information representation.

[0869] "Emotional identification" is a technology that identifies the emotional nuances behind input voice or text, and is a process for estimating the user's emotional state.

[0870] A "generative AI model" is a system that utilizes artificial intelligence to produce creative responses and outputs based on input data, and is a model designed to provide information that is appropriate to emotions and context.

[0871] A "translation model" is an algorithm or mechanism for converting text from one language to another, and it is a technology that acts as a bridge for conveying meaning between different languages.

[0872] A "designated symbolic system" refers to the target language or form of expression, and is a set of languages ​​and symbols defined in advance as the target of translation or processing.

[0873] "Speech synthesis technology" is a technology that artificially generates acoustic signals based on text data and outputs them as natural-sounding speech. It is a process that converts textual information into a form that humans can recognize by hearing.

[0874] "Audio data" refers to the digital format of sound generated by speech synthesis technology, and is the final output information provided to the user.

[0875] A "user" is a person or organization that receives services provided by the system, and is the entity that receives translation and information through interaction with the system.

[0876] As an embodiment of this invention, the system is constructed as follows. This system processes voice, text, and image input and provides real-time translation that takes emotion into account.

[0877] First, the user inputs voice into the device. The device captures the voice using its microphone and performs preprocessing such as noise reduction and normalization. Next, the device uses speech recognition technology (for example, the Google Speech-to-Text API) to convert the voice into text and sends that data to the server.

[0878] The server receives text data and uses an emotion analysis engine (for example, IBM Watson Tone Analyzer) to recognize emotions. This engine estimates the user's emotions from the tone of voice and nuances of speech, and passes that information to a generative AI model.

[0879] The generative AI model receives text to be translated based on sentiment information and uses a translation model (e.g., Google Translate API) to perform a translation that reflects natural and appropriate sentiment in the specified language. The translation result is then converted back into audio data using speech synthesis technology (e.g., Amazon Polly).

[0880] Finally, the device plays this audio data through its speaker and provides it to the user. This allows the user to receive the translated result, which reflects emotions, as audio.

[0881] As a concrete example, when a home robot conveys the gentle words of a parent who speaks Japanese to a child learning English, the voice assistant can analyze the parent's tone using an emotion engine, generate English with a positive nuance using an AI model, and provide a translation result that gives the child a sense of security.

[0882] As an example of a prompt, by inputting a request such as, "When Mom asks the robot, 'How was your homework today?', please convey the message in a gentle tone," the system can generate output corresponding to that instruction.

[0883] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0884] Step 1:

[0885] The user inputs voice into the device's microphone. The device captures the voice and performs initial preprocessing. This preprocessing applies a noise reduction filter and normalizes the audio signal, preparing it for high-quality data to be sent to the server. The input is raw audio data, and the output is normalized audio data with noise removed.

[0886] Step 2:

[0887] The device converts speech to text by sending pre-processed audio data to a speech recognition engine (e.g., Google Speech-to-Text API). The speech recognition engine analyzes the acoustic characteristics and generates corresponding string data. The input is de-noised audio data, and the output is the converted text data.

[0888] Step 3:

[0889] The terminal transfers the generated text data to the server. The server receives the text data and analyzes the sentiment information using a sentiment analysis engine (e.g., IBM Watson Tone Analyzer). The sentiment analysis engine estimates the user's emotions from the structure and word choice of the text and outputs it as an emotion vector. The input is text data, and the output is an emotion vector.

[0890] Step 4:

[0891] The server sends emotion vectors to a generating AI model, which then passes them along with prompt text to a translation model (e.g., Google Translate API). This model considers the emotion information and translates the text into the specified language while maintaining appropriate tone and wording. The input is text data and emotion vectors, and the output is the translated text.

[0892] Step 5:

[0893] The server passes the translated text to a speech synthesis technology (e.g., Amazon Polly) to generate audio data. The speech synthesis engine synthesizes speech with natural tones and intonations that reflect emotional information, and outputs it as digital audio data. The input is the translated text, and the output is the synthesized audio data.

[0894] Step 6:

[0895] The terminal provides the user with audio data received from the server by playing it through its speaker. By listening to this, the user can receive a translation that reflects emotions. The input is synthesized audio data, and the output is the played audio.

[0896] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0897] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0898] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0899] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0900] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0901] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0902] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0903] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0904] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0905] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0906] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0907] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0908] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0909] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0910] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0911] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0912] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0913] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0914] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0915] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0916] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0917] The following is further disclosed regarding the embodiments described above.

[0918] (Claim 1)

[0919] A means for receiving voice input, performing preprocessing, and converting the voice into text using speech recognition technology,

[0920] A means of translating text converted using a translation model into a specified language,

[0921] A means of converting translated text into audio data using speech synthesis technology,

[0922] Means for providing the audio data to the user,

[0923] A system that includes this.

[0924] (Claim 2)

[0925] The system according to claim 1, comprising means for receiving text input, performing preprocessing, and translating the text into a specified language using a translation model.

[0926] (Claim 3)

[0927] The system according to claim 1, comprising means for receiving image data, extracting text information from the image using optical character recognition technology, and translating the extracted text into a specified language using a translation model.

[0928] "Example 1"

[0929] (Claim 1)

[0930] A means of acquiring audio data, performing preprocessing using data processing technology, and converting it into text data using speech recognition technology,

[0931] A method for translating text data into a target language using a generative AI model,

[0932] A means of converting translated text data into a digital audio format and outputting it as audio using presentation technology,

[0933] An integrated platform that centrally processes input formats selected by the user from voice, text, and images, and provides multilingual translation.

[0934] A system that includes this.

[0935] (Claim 2)

[0936] The system according to claim 1, which receives text information and performs a process of translating the text information into a predetermined language using a generating AI model.

[0937] (Claim 3)

[0938] The system according to claim 1, which acquires image information, extracts text information from the image using optical character recognition technology, and performs a process of translating the text information into a target language using a generative AI model.

[0939] "Application Example 1"

[0940] (Claim 1)

[0941] A means for receiving voice input, performing preprocessing, and converting the voice into text using speech recognition technology,

[0942] A means for translating text converted using a translation model into a specified language and converting it into audio data using speech synthesis technology,

[0943] A means for processing image data, extracting character information from the image using optical character recognition technology, and displaying the translated result on the image using a translation model,

[0944] Means of providing information to support multilingual travel,

[0945] A system that includes this.

[0946] (Claim 2)

[0947] The system according to claim 1, comprising means for receiving text input, performing preprocessing, translating the text into a specified language using a translation model, and displaying the result.

[0948] (Claim 3)

[0949] The system according to claim 1, comprising means for providing multilingual support using audio and image information and overcoming the language barrier for the user.

[0950] "Example 2 of combining an emotion engine"

[0951] (Claim 1)

[0952] A means for receiving voice input, performing preprocessing, and converting the voice into text information using speech recognition technology,

[0953] A means of analyzing the sentiment of text information converted using sentiment analysis technology,

[0954] A method that takes emotional information into account, uses a translation model to convert text information into a specified language, and modifies the wording and tone to match the emotions.

[0955] A means of converting translated text information into audio data using speech synthesis technology,

[0956] Means for providing the audio data to the user,

[0957] A system that includes this.

[0958] (Claim 2)

[0959] The system according to claim 1, comprising means for receiving text information, analyzing the sentiment of the text information using sentiment analysis technology, and converting the text information into a specified language using a translation model based on the sentiment information.

[0960] (Claim 3)

[0961] The system according to claim 1, comprising means for receiving image information, extracting text information from the image information using optical character recognition technology, analyzing the sentiment of the extracted text information and visual elements in the image using sentiment analysis technology, and reflecting the results in a translation model to convert into a specified language.

[0962] "Application example 2 when combining with an emotional engine"

[0963] (Claim 1)

[0964] A means for receiving audio input, performing preprocessing, and converting the audio into a sequence of symbols using acoustic recognition technology,

[0965] A means of identifying emotions and passing the estimated emotion information to a generating AI model,

[0966] A means of translating a sequence of symbols converted using a translation model into a specified symbol system, and adjusting the wording and tone based on emotional information,

[0967] A means for converting a translated sequence of symbols into audio data using speech synthesis technology,

[0968] Means for providing the audio data to the user,

[0969] A system that includes this.

[0970] (Claim 2)

[0971] The system according to claim 1, comprising means for receiving a sequence of symbols as input, performing preprocessing, analyzing emotions, passing the emotional information to a generating AI model, and translating the sequence of symbols into a specified symbolic system using a translation model.

[0972] (Claim 3)

[0973] The system according to claim 1, comprising means for receiving visual data, extracting symbolic information from the visual data using optical character recognition technology, analyzing emotions, passing the emotion information to a generating AI model, and translating the extracted symbols into a specified symbolic system using a translation model. [Explanation of symbols]

[0974] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving voice input, performing preprocessing, and converting the voice into text using speech recognition technology, A means of translating text converted using a translation model into a specified language, A means of converting translated text into audio data using speech synthesis technology, Means for providing the audio data to the user, A system that includes this.

2. The system according to claim 1, comprising means for receiving text input, performing preprocessing, and translating the text into a specified language using a translation model.

3. The system according to claim 1, comprising means for receiving image data, extracting text information from the image data using optical character recognition technology, and translating the extracted text into a specified language using a translation model.

Images