system

Abstract

We provide the system. [Solution] A means of acquiring voice data from the user, A conversion means for converting acquired audio data into text data, Translation methods for translating text data into multiple languages, A transmission means for sending translated text data to the user's terminal, An output means for outputting the translation result to the user in audio or text, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a multi-cultural society, it is necessary to alleviate communication barriers between people speaking different languages and enable foreigners to live without feeling socially isolated in a new environment. However, conventional translation systems are difficult to convey accurate nuances and are often not suitable for real-time interactions. Therefore, a more innovative system for facilitating cross-cultural communication is required.

Means for Solving the Problems

[0005] This invention provides a technology that converts voice data acquired from a user into text data and translates the converted text data into multiple languages. To achieve this, it combines conversion and translation means and transmits the translated data to the user's terminal, thereby supporting cross-cultural communication. Furthermore, by using a generative model, it enables natural dialogue that takes linguistic nuances into account during translation. In addition, by receiving user feedback and continuously improving the translation algorithm based on that feedback, it provides a highly accurate system.

[0006] "Acquisition means" refers to the means of receiving and processing voice data from the user.

[0007] A "conversion means" is a means for converting acquired audio data into text data.

[0008] A "translation method" is a means of translating converted text data into multiple languages.

[0009] "Transmission means" refers to the means for sending translated text data to the user's terminal.

[0010] "Output means" refers to a means of providing the user with the translation result in audio or text format.

[0011] A "generative model" is an algorithm or method for generating natural-sounding translations while taking into account the nuances of language.

[0012] "Learning methods" refer to means of using user feedback information to improve translation algorithms. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applicable to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system specifically designed to facilitate communication between people who speak different languages ​​in a multicultural society. The system is designed around multilingual translation and interpretation functions to support user communication in real time.

[0035] The user provides voice input to the device in their own language. This voice data is captured by an acquisition mechanism within the device. Once the voice data is acquired, the device automatically converts it into text data using a conversion mechanism. This process utilizes natural language processing technology.

[0036] The server receives the converted text data and translates it into the specified target language using a translation tool. This translation process utilizes a generative model to achieve natural and fluent translations that are contextually accurate, rather than simply word-for-word matching. Once the translation is complete, the server sends the translated text to the terminal.

[0037] The device uses the received translated text to output its content to the user in either voice or text format. If voice output is requested, the output device within the device converts the text into voice data and conveys the translated content to the user.

[0038] As a concrete example, if a user says "What would you like to eat today?" in Japanese, the device captures this and converts it to text. The server receives this text, translates it into English, and sends the translated result, "What would you like to eat today?", back to the device. The device then plays this English sentence back to the user as audio, enabling smooth communication with someone who speaks a different language.

[0039] Furthermore, the system receives feedback from users, and the server uses this information to improve the translation algorithm through learning mechanisms. This iterative learning process continuously improves the overall translation accuracy of the system.

[0040] In this way, the present invention provides an effective means of enabling free and natural communication among people who speak different languages ​​and reducing barriers to dialogue in multicultural societies.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The user provides voice input in their own language to the device. This voice data is captured by the device's built-in acquisition mechanism.

[0044] Step 2:

[0045] The terminal passes the captured audio data to a conversion device, which then converts the audio data into text data. This conversion is performed using speech recognition technology.

[0046] Step 3:

[0047] The converted text data is sent from the terminal to the server. The server then passes the received text data to the translation device.

[0048] Step 4:

[0049] The server translates text data into the specified target language using translation tools. A generative model is used to produce a natural translation that takes linguistic nuances into account.

[0050] Step 5:

[0051] The translated text data is sent from the server to the terminal. The terminal then prepares the data for easy viewing and guidance.

[0052] Step 6:

[0053] The device outputs the received translated text to the user. If voice output is selected, it uses text-to-speech technology to output the translation result to the user.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] In multicultural societies, language barriers exist in communication between speakers of different languages. This makes real-time communication difficult, and there is a need for means to facilitate smooth communication based on mutual understanding of each other's cultural backgrounds and linguistic characteristics.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes a device for acquiring acoustic information from a user, a device for converting the acquired acoustic information into symbolic information, a device for translating the symbolic information into another language, a device for transmitting the translated symbolic information to the user's information terminal, and a device for outputting the translation result to the user in sound or symbols. This makes it possible to provide natural and fluent translations that take into account context and cultural nuances between users of different languages, and to support real-time communication.

[0059] "Acoustic information" refers to data recorded by mechanical or electronic means from sound transmitted as vibrations in the air.

[0060] An "information terminal" is an electronic device that processes voice and symbolic data and provides information to the user.

[0061] "Symbolic information" refers to string data that represents language or content according to specific rules.

[0062] A "generative algorithm" is a computational procedure or model for generating new data from existing data.

[0063] "Evaluation information" refers to reactions and opinions collected from users regarding specific operations or functions.

[0064] A "device that outputs sound or symbols" is a device that presents information to the user in audio or visual form.

[0065] A "device for translating into other languages" is a device that has the function of converting information expressed in one language into another language.

[0066] This invention is a system that facilitates communication between users who speak different languages ​​in a multicultural society. The system supports natural and fluent communication by translating the user's speech into other languages ​​in real time.

[0067] When a user speaks a different language, they input audio information into the device in their native language. The device is equipped with a microphone to capture sound, thereby acquiring the audio information. The devices used are typical smartphones and tablets, and by installing speech recognition software (e.g., commercial speech recognition applications) on them, the speech is converted into symbolic information.

[0068] The server receives symbolic information sent from the terminal and uses a generation algorithm to translate it into other languages, taking into account the context and nuances of the language. This translation utilizes a multilingual translation API or a generative AI model (e.g., a general-purpose generative AI platform). The generative AI model learns from user feedback to improve translation accuracy.

[0069] The terminal receives translated symbolic information sent from the server and presents the information according to the output format (audio or symbol) selected by the user. Audio output utilizes speech synthesis software, while symbolic output utilizes screen display functionality. Depending on the settings, the user may experience either audio playback or text display.

[0070] As a concrete example, if user A speaks in Japanese, "What would you like to eat today?", the terminal captures this audio and converts it into symbolic information. The server receives this information and generates the text "What would you like to eat today?" translated into English. The generated text is sent to the terminal, which then outputs this text as speech to user B in English.

[0071] Examples of prompt statements include the following:

[0072] "Translate the Japanese text input by User A via voice input into English, a language that User B understands."

[0073] "To improve translation accuracy, please use user feedback to adjust the parameters of the generative AI model."

[0074] In this way, the system enables communication that takes cultural backgrounds and contexts into account between people who speak different languages.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The user speaks into the device in their own language, inputting audio data as sound information. The device captures this audio using its built-in microphone and temporarily stores it as a digital audio file. At this stage, noise reduction and audio clearing are performed to ensure high-quality input data.

[0078] Step 2:

[0079] The device converts acquired acoustic information into symbolic information (text data) using speech recognition software. The main processing includes phoneme analysis through digital signal processing and word recognition using a language model. The output is the converted audio text, ready for transmission to the cloud server.

[0080] Step 3:

[0081] The terminal sends the converted text data to the server. An efficient protocol is used for transmission to minimize network latency. The server receives this text information and formats the data so that it can be input into the multilingual translation API.

[0082] Step 4:

[0083] The server translates the received symbolic information into the target language using a generation AI model. This process utilizes contextual analysis and natural language processing to understand linguistic nuances and generate appropriate translation results. The output is translated text data, which is then prepared for return to the terminal.

[0084] Step 5:

[0085] The terminal receives the translation results sent from the server. The received text is output to the user in the specified format (sound or symbols). In the case of sound output, speech synthesis software is used to convert the text back into speech and play it through the speaker. If output as symbols, the text is displayed on the screen.

[0086] Step 6:

[0087] Users provide feedback on the system's translation results. They input evaluation information on their device and send it to the server. The server analyzes this feedback and uses it as new training data to retrain the AI ​​model, thereby improving the translation algorithm. Through this learning process, the system's translation accuracy continuously improves.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] This addresses the challenge of facilitating smooth communication in home environments where people speak different languages. In multicultural societies, technologies that support dialogue between people who speak different languages ​​are needed to improve the quality of life.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes an acquisition means for acquiring voice information from a user, a conversion means for converting the acquired voice information into text information, a translation means for converting the text information into multiple languages, a transmission means for delivering the translated text information to the user's device, an output means for presenting the translation results to the user in voice or text, a feedback means for collecting user responses and improving translation accuracy, and a dialogue means adapted to smart devices and supporting intercultural communication in a home environment. This enables smooth intercultural dialogue.

[0093] "Acquisition means" refers to a device or function for acquiring voice information from a user.

[0094] "Conversion means" refers to a function that converts acquired audio information into text information.

[0095] "Translation means" refers to technologies or algorithms for converting text information into multiple languages.

[0096] "Transmission means" refers to the technology for delivering translated text information to the user's device.

[0097] "Output method" refers to a function that presents the translation result to the user in audio or text format.

[0098] A "feedback mechanism" is a technology used to collect user feedback and improve translation accuracy.

[0099] "Dialogue tools" refer to functions adapted to smart devices that support intercultural exchange in the home environment.

[0100] This invention realizes a system using a smart device that enables individual users who speak different languages ​​to communicate smoothly. The system mainly consists of a voice acquisition means, a conversion means, a translation means, a transmission means, an output means, a feedback means, and a dialogue means.

[0101] The server uses the microphone in the device to obtain voice information from the user. This voice information is converted into text information using a speech recognition engine installed on the server or terminal (e.g., Google® Cloud Speech-to-Text). The converted text is then translated into multiple target languages ​​by a translation tool. The translation uses a generative model that utilizes natural language processing (e.g., Google Translate API) to provide contextually natural translations.

[0102] The translated text information is sent from the server to the user's terminal and can be output in audio format using a speech synthesis engine (e.g., Amazon Polly). During this process, user responses are collected as feedback, which the server analyzes to improve the accuracy of the translation algorithm.

[0103] For example, when offering tea to a foreign guest at home, if the user says "Would you like some tea?" in Japanese, the system will translate and output this in real time as "Would you like some tea?" in English, enabling smooth communication.

[0104] An example of a prompt using a generative AI model is, "Explain how smart devices can improve dialogue between different languages." In this way, the system provides a means to facilitate free and natural dialogue in multicultural environments and lower language barriers.

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The user speaks into the device's microphone. This input audio data is captured by the terminal. The captured audio data undergoes digital signal processing to reduce noise and optimize it.

[0108] Step 2:

[0109] The device sends the captured audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The input is audio data, and the output is text data that reflects its content. This conversion process uses speech signal processing algorithms to analyze and map phonemes.

[0110] Step 3:

[0111] The server receives text data from the terminal and sends it to the translation tool, where it is converted into multiple languages. A generative model (e.g., Google Translate API) receives the input text and translates it into the target language. The data calculations performed here involve natural language processing and string reconstruction based on contextual awareness.

[0112] Step 4:

[0113] The server sends the translated text data to the terminal. The terminal inputs the received translated text into a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The output is the translated speech data. Speech synthesis involves the process of generating a sequence of phonemes from the text and constructing a speech waveform.

[0114] Step 5:

[0115] The device plays the translation results to the user through its speaker. In this step, adjustments are made to ensure that the information is delivered to the user at an appropriate volume and clarity.

[0116] Step 6:

[0117] User reactions and feedback are recorded by the device and sent to the server. The server analyzes the feedback information and uses it to improve the translation algorithm. Here, data mining techniques are used to evaluate translation quality and adjust the algorithm.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] This invention provides a system that combines an emotion engine to further enhance multilingual communication. This system enables users to communicate smoothly in different language environments and to engage in accurate and emotionally sensitive communication.

[0120] The user makes voice input into the device and speaks with emotion. This voice data is captured by the device's acquisition mechanism. The voice data is first converted into text data by a conversion mechanism. Along with the converted text, the voice data is supplied to an emotion engine, which analyzes the user's emotional state.

[0121] The server uses the sentiment information obtained through this analysis to generate expressions that reflect the user's emotions when translating text data into multiple languages ​​using a translation tool. The generative model provides more natural and emotionally conveying translation results by considering the linguistic nuances based on that sentiment information.

[0122] The device then outputs a translation to the user that takes these generated emotions into consideration. For example, if the emotion is positive, the expression is adjusted to be more approachable; if it is negative, adjustments are made such as adding carefully follow-up content.

[0123] As a concrete example, consider a scenario where a user expresses anger by saying, "Why did this happen?" The device sends this to the emotion engine, and the server, based on the emotional information indicating anger, can translate it as "Why did this happen in such a way?" and add empathetic phrases such as "I understand your frustration."

[0124] This system also features a feedback function that analyzes user feedback to further refine its translation algorithms and sentiment recognition capabilities. This allows for continuous improvement of the overall system performance and enables the provision of a better user experience.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] The user inputs voice data into the terminal. The terminal receives this voice data using an acquisition method.

[0128] Step 2:

[0129] The terminal converts the received audio data into text data using a conversion mechanism. Speech recognition technology is used in this process.

[0130] Step 3:

[0131] The device sends the converted text data to the server. Simultaneously, the voice data is sent to the emotion engine for sentiment analysis.

[0132] Step 4:

[0133] The server adjusts the translation content using sentiment information obtained from the sentiment engine during the process of translating text data into the target language specified by the translation means. A generative model is utilized to achieve natural translations that reflect emotions.

[0134] Step 5:

[0135] The server sends the translated text data to the user's device.

[0136] Step 6:

[0137] The device outputs the received translated text as either audio or text. Depending on the selection, text-to-speech technology is used to deliver an emotionally sensitive translation to the user.

[0138] (Example 2)

[0139] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0140] In a multilingual environment, achieving natural communication that appropriately reflects user emotions is challenging. Traditional translation systems often fail to consider user emotions, resulting in a lack of nuance and misunderstandings in communication. Furthermore, mechanisms for improving translation quality through user feedback have been insufficient.

[0141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0142] In this invention, the server includes means for analyzing information, means for translating the information into multiple languages ​​using the analysis results, and means for receiving feedback information and using it to improve the algorithm. This enables natural and accurate translation that reflects the user's emotions and allows for continuous performance improvement based on feedback.

[0143] "Information" refers to audio data obtained from users and the resulting text data.

[0144] "Means of acquisition" refers to devices or systems for collecting user speech as audio data.

[0145] "Means of conversion" refers to the processes and technologies used to convert collected audio data into text data.

[0146] "Means of analysis" refers to technical methods for analyzing the user's emotional state based on the converted text data.

[0147] "Methods for multilingual translation" refer to processes and models for translating text data into different languages ​​based on analysis results.

[0148] "Means of transmission" refers to communication technology used to transmit translated information to the user's device.

[0149] "Means of output" refers to devices or software that display the translation results in audio or text format to communicate them to the user.

[0150] A "generative model" is an algorithm or program that provides translations while taking into account the nuances and emotional information of language.

[0151] "Feedback information" refers to the evaluations and opinions that users provide regarding the translation results.

[0152] "Algorithm improvement" is the process of improving system performance by utilizing the feedback information received.

[0153] This invention is a system that enhances user communication in a multilingual environment. This system provides translations that take user emotions into consideration, enabling smooth and emotionally sensitive communication.

[0154] The user speaks into the device, and the resulting audio is input. Suitable hardware includes the built-in microphone of a smartphone or tablet, or an external microphone device.

[0155] The device uses speech recognition software to convert this speech data into text data. Specific software examples include "Google Cloud Speech-to-Text" and "Microsoft® Azure® Speech Service."

[0156] The converted text data and original audio are sent to the emotion engine. The emotion engine analyzes the user's emotional state using tools such as "IBM Watson® Tone Analyzer".

[0157] The server uses the analyzed sentiment information to translate text data into other languages ​​via a generative AI model. In this process, generative AIs such as "OpenAI® GPT-3®" and "DeepL" are used to generate expressions that take into account linguistic nuances and hierarchical relationships.

[0158] The translation results are sent to the user's device using the appropriate communication technology. The device then outputs the received translation results to the user via speech synthesis technology or screen display. This ensures that the translation results are conveyed to the user in a natural way that reflects emotional information. For example, if the translation results indicate a negative emotion, a follow-up phrase such as "I understand your frustration" may be added.

[0159] A feedback function is also provided, allowing users to submit opinions and ratings regarding translations. This feedback is used to improve the system's algorithms.

[0160] An example of a prompt is: "Generate an English translation for when the user is angry." Using this prompt, the generative AI model will provide a translation that takes the user's emotions into account.

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The user speaks into the device. The user's voice is input and captured by the device's microphone. This input voice contains the user's intentions and emotions. In this step, the voice data is taken into the system.

[0164] Step 2:

[0165] The device uses speech recognition software to convert the input speech data into text data. Here, the speech signal is analyzed and a corresponding string of characters is generated. This conversion is performed using speech recognition technology such as "Google Cloud Speech-to-Text," and the output is text data.

[0166] Step 3:

[0167] The device passes the generated text data and the original audio data to the emotion engine. This engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. This analysis uses tools such as the "IBM Watson Tone Analyzer." The input consists of text data and audio data, and the output is emotion information.

[0168] Step 4:

[0169] The server uses emotional information and text data received from the emotion engine to perform multilingual translation. During this process, generative AI models such as "OpenAI GPT-3" and "DeepL" are used to generate expressions that reflect emotions. This process outputs translated data that incorporates emotional information.

[0170] Step 5:

[0171] The terminal receives translation results sent from the server and outputs them to the user. Output is done through text display or speech synthesis. Specifically, the user is presented with translation results that include expressions that reflect emotions.

[0172] Step 6:

[0173] Users provide feedback on the translation results. This feedback is sent from the terminal to the server as needed and used as data for further system improvements. In this step, feedback information is collected and used to improve the algorithm.

[0174] (Application Example 2)

[0175] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0176] In the multilingual environments we encounter in our daily lives, it's not enough to simply translate languages; we also need to accurately convey the speaker's emotions. However, conventional translation technologies lack the ability to consider emotions, resulting in sluggish communication. Therefore, achieving natural and emotionally rich communication in multiple languages ​​is a challenge.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0178] In this invention, the server includes an emotion analysis means for analyzing the user's emotional state, a generation means for generating translation results that reflect the emotional state, and a means for the translation means to consider the nuances of language using a generation model and add expressions based on emotional information. This makes it possible to provide translations that take the user's emotions into consideration and improve the quality of communication.

[0179] "Acquisition means" refers to a device or function for collecting voice data from a user.

[0180] "Conversion means" refers to a device or function for converting acquired audio data into text data.

[0181] "Translation means" refers to a device or function for translating text data into multiple languages.

[0182] "Emotional analysis means" refers to a device or function for analyzing a user's emotional state.

[0183] "Generating means" refers to a device or function for generating translation results that reflect emotional states.

[0184] "Transmission means" refers to a device or function for transferring the generated translation data to the user's terminal.

[0185] "Output means" refers to a device or function for presenting the translation result to the user in audio or text format.

[0186] A "generative model" is an artificial intelligence model used for translation and text generation that takes into account the nuances of language in natural language processing.

[0187] A specific embodiment for implementing this invention will be described. The user inputs to the terminal using voice and obtains voice data. The terminal recognizes the speech with emotion and collects the voice data by an acquisition means. Next, the terminal converts the voice data into text data using a conversion means. This converted text and voice data are transmitted to a server, and the emotional state of the user is analyzed by an emotion analysis means.

[0188] The server uses the analyzed emotion information to generate a translation result reflecting the emotional state through a generation means. A generative AI model is used for the translation means, and a translation considering the nuances of language and emotion information is performed. At this time, additional expressions are adjusted based on the emotion information. The generated translation data is transmitted to the terminal via a transmission means and finally presented to the user in voice or text by an output means.

[0189] As an example, when the user says "I'm very tired today...", the system recognizes the emotion and adds an emotionally rich expression such as "I hope you can relax and enjoy your meal!" when translating it into English. Also, as a specific example of a prompt sentence, "Please generate a translation of the Japanese sentence '今日はとても疲れていて…' into English, considering the sentiment to be positive and adding an encouraging follow-up. Use the emotional tone to craft a naturally flowing response." can be cited.

[0190] In this way, the system enhances multilingual emotional communication with users, enabling natural and smooth dialogue.

[0191] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0192] Step 1:

[0193] The user inputs voice through the microphone. This voice data is acquired by the device. The voice input is natural and concrete speech and is transmitted to the device as audio waveform data.

[0194] Step 2:

[0195] The terminal converts the acquired audio data into text data using a speech recognition engine. Audio waveform data is input, analyzed by the speech recognition engine, and output as text data in string format.

[0196] Step 3:

[0197] The terminal sends the converted text and audio data to the server. The data sent is text data in string format and audio waveform data.

[0198] Step 4:

[0199] The server inputs the received text data into a sentiment analysis device to analyze the user's emotional state. After the text data is analyzed, metadata indicating the user's emotions is output.

[0200] Step 5:

[0201] The server uses emotional metadata to translate text data into multiple languages ​​using a generative AI model and translation tools. The input data consists of text data and emotional metadata, and the output is translated text that takes emotions into account.

[0202] Step 6:

[0203] The server adds emotion-based additional expressions to the translated text through a generation mechanism. The input is the translated text, and the output is the adjusted translated text that reflects the emotion.

[0204] Step 7:

[0205] The server sends the generated translated text to the terminal. This transmitted data is an adjusted translated text that takes sentiment into account.

[0206] Step 8:

[0207] The terminal presents the received translated text to the user using an output device. The user receives the final translation result in either audio or text format.

[0208] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0209] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0210] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0211] [Second Embodiment]

[0212] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0213] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0214] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0215] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0216] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0217] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0218] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0219] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0220] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0221] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0222] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0223] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0224] This invention is a system specifically designed to facilitate communication between people who speak different languages ​​in a multicultural society. The system is designed around multilingual translation and interpretation functions to support user communication in real time.

[0225] The user provides voice input to the device in their own language. This voice data is captured by an acquisition mechanism within the device. Once the voice data is acquired, the device automatically converts it into text data using a conversion mechanism. This process utilizes natural language processing technology.

[0226] The server receives the converted text data and translates it into the specified target language using a translation tool. This translation process utilizes a generative model to achieve natural and fluent translations that are contextually accurate, rather than simply word-for-word matching. Once the translation is complete, the server sends the translated text to the terminal.

[0227] The device uses the received translated text to output its content to the user in either voice or text format. If voice output is requested, the output device within the device converts the text into voice data and conveys the translated content to the user.

[0228] As a concrete example, if a user says "What would you like to eat today?" in Japanese, the device captures this and converts it to text. The server receives this text, translates it into English, and sends the translated result, "What would you like to eat today?", back to the device. The device then plays this English sentence back to the user as audio, enabling smooth communication with someone who speaks a different language.

[0229] Furthermore, the system receives feedback from users, and the server uses this information to improve the translation algorithm through learning mechanisms. This iterative learning process continuously improves the overall translation accuracy of the system.

[0230] In this way, the present invention provides an effective means of enabling free and natural communication among people who speak different languages ​​and reducing barriers to dialogue in multicultural societies.

[0231] The following describes the processing flow.

[0232] Step 1:

[0233] The user provides voice input in their own language to the device. This voice data is captured by the device's built-in acquisition mechanism.

[0234] Step 2:

[0235] The terminal passes the captured audio data to a conversion device, which then converts the audio data into text data. This conversion is performed using speech recognition technology.

[0236] Step 3:

[0237] The converted text data is sent from the terminal to the server. The server then passes the received text data to the translation device.

[0238] Step 4:

[0239] The server translates text data into the specified target language using translation tools. A generative model is used to produce a natural translation that takes linguistic nuances into account.

[0240] Step 5:

[0241] The translated text data is sent from the server to the terminal. The terminal then prepares the data for easy viewing and guidance.

[0242] Step 6:

[0243] The device outputs the received translated text to the user. If voice output is selected, it uses text-to-speech technology to output the translation result to the user.

[0244] (Example 1)

[0245] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0246] In multicultural societies, language barriers exist in communication between speakers of different languages. This makes real-time communication difficult, and there is a need for means to facilitate smooth communication based on mutual understanding of each other's cultural backgrounds and linguistic characteristics.

[0247] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0248] In this invention, the server includes a device for acquiring acoustic information from a user, a device for converting the acquired acoustic information into symbolic information, a device for translating the symbolic information into another language, a device for transmitting the translated symbolic information to the user's information terminal, and a device for outputting the translation result to the user in sound or symbols. This makes it possible to provide natural and fluent translations that take into account context and cultural nuances between users of different languages, and to support real-time communication.

[0249] "Acoustic information" refers to data recorded by mechanical or electronic means from sound transmitted as vibrations in the air.

[0250] An "information terminal" is an electronic device that processes voice and symbolic data and provides information to the user.

[0251] "Symbolic information" refers to string data that represents language or content according to specific rules.

[0252] A "generative algorithm" is a computational procedure or model for generating new data from existing data.

[0253] "Evaluation information" refers to reactions and opinions collected from users regarding specific operations or functions.

[0254] A "device that outputs sound or symbols" is a device that presents information to the user in audio or visual form.

[0255] A "device for translating into other languages" is a device that has the function of converting information expressed in one language into another language.

[0256] This invention is a system that facilitates communication between users who speak different languages ​​in a multicultural society. The system supports natural and fluent communication by translating the user's speech into other languages ​​in real time.

[0257] When a user speaks a different language, they input audio information into the device in their native language. The device is equipped with a microphone to capture sound, thereby acquiring the audio information. The devices used are typical smartphones and tablets, and by installing speech recognition software (e.g., commercial speech recognition applications) on them, the speech is converted into symbolic information.

[0258] The server receives symbolic information sent from the terminal and uses a generation algorithm to translate it into other languages, taking into account the context and nuances of the language. This translation utilizes a multilingual translation API or a generative AI model (e.g., a general-purpose generative AI platform). The generative AI model learns from user feedback to improve translation accuracy.

[0259] The terminal receives translated symbolic information sent from the server and presents the information according to the output format (audio or symbol) selected by the user. Audio output utilizes speech synthesis software, while symbolic output utilizes screen display functionality. Depending on the settings, the user may experience either audio playback or text display.

[0260] As a concrete example, if user A speaks in Japanese, "What would you like to eat today?", the terminal captures this audio and converts it into symbolic information. The server receives this information and generates the text "What would you like to eat today?" translated into English. The generated text is sent to the terminal, which then outputs this text as speech to user B in English.

[0261] Examples of prompt statements include the following:

[0262] "Translate the Japanese text input by User A via voice input into English, a language that User B understands."

[0263] "To improve translation accuracy, please use user feedback to adjust the parameters of the generative AI model."

[0264] In this way, the system enables communication that takes cultural backgrounds and contexts into account between people who speak different languages.

[0265] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0266] Step 1:

[0267] The user speaks into the device in their own language, inputting audio data as sound information. The device captures this audio using its built-in microphone and temporarily stores it as a digital audio file. At this stage, noise reduction and audio clearing are performed to ensure high-quality input data.

[0268] Step 2:

[0269] The device converts acquired acoustic information into symbolic information (text data) using speech recognition software. The main processing includes phoneme analysis through digital signal processing and word recognition using a language model. The output is the converted audio text, ready for transmission to the cloud server.

[0270] Step 3:

[0271] The terminal sends the converted text data to the server. An efficient protocol is used for transmission to minimize network latency. The server receives this text information and formats the data so that it can be input into the multilingual translation API.

[0272] Step 4:

[0273] The server translates the received symbolic information into the target language using a generation AI model. This process utilizes contextual analysis and natural language processing to understand linguistic nuances and generate appropriate translation results. The output is translated text data, which is then prepared for return to the terminal.

[0274] Step 5:

[0275] The terminal receives the translation results sent from the server. The received text is output to the user in the specified format (sound or symbols). In the case of sound output, speech synthesis software is used to convert the text back into speech and play it through the speaker. If output as symbols, the text is displayed on the screen.

[0276] Step 6:

[0277] Users provide feedback on the system's translation results. They input evaluation information on their device and send it to the server. The server analyzes this feedback and uses it as new training data to retrain the AI ​​model, thereby improving the translation algorithm. Through this learning process, the system's translation accuracy continuously improves.

[0278] (Application Example 1)

[0279] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0280] This addresses the challenge of facilitating smooth communication in home environments where people speak different languages. In multicultural societies, technologies that support dialogue between people who speak different languages ​​are needed to improve the quality of life.

[0281] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0282] In this invention, the server includes an acquisition means for acquiring voice information from a user, a conversion means for converting the acquired voice information into text information, a translation means for converting the text information into multiple languages, a transmission means for distributing the translated text information to the user's device, an output means for presenting the translation result to the user in voice or text, a feedback means for collecting the user's reaction and improving the translation accuracy, and a dialogue means adapted to a smart device and assisting cross-cultural communication in a home environment. Thereby, smooth dialogue between different cultures becomes possible.

[0283] The "acquisition means" is a device or function for acquiring voice information from a user.

[0284] The "conversion means" refers to the function of converting the acquired voice information into text information.

[0285] The "translation means" is a technology or algorithm for converting text information into multiple languages.

[0286] The "transmission means" is a technology for distributing the translated text information to the user's device.

[0287] The "output means" refers to the function of presenting the translation result to the user in voice or text.

[0288] The "feedback means" is a technology for collecting the user's reaction and improving the translation accuracy.

[0289] The "dialogue means" is a function adapted to a smart device and assisting cross-cultural communication in a home environment.

[0290] This invention realizes a system using a smart device that enables individual users speaking different languages to communicate smoothly. The system mainly consists of a voice acquisition means, a conversion means, a translation means, a transmission means, an output means, a feedback means, and a dialogue means. <00009**19**> It seems there is a small error in the original text where the number in tag

[0291] might be incorrect as it's shown as <000********19> in the original. I've translated it as

[0291] in the English version according to the pattern. If this is not what you intended, please correct the original text and let me know.The server uses the microphone in the device to obtain voice information from the user. This voice information is converted into text information using a speech recognition engine installed on the server or terminal (e.g., Google Cloud Speech-to-Text). The converted text is then translated into multiple target languages ​​by a translation tool. The translation uses a generative model that utilizes natural language processing (e.g., Google Translate API) to provide contextually natural translations.

[0292] The translated text information is sent from the server to the user's terminal and can be output in audio format using a speech synthesis engine (e.g., Amazon Polly). During this process, user responses are collected as feedback, which the server analyzes to improve the accuracy of the translation algorithm.

[0293] For example, when offering tea to a foreign guest at home, if the user says "Would you like some tea?" in Japanese, the system will translate and output this in real time as "Would you like some tea?" in English, enabling smooth communication.

[0294] An example of a prompt using a generative AI model is, "Explain how smart devices can improve dialogue between different languages." In this way, the system provides a means to facilitate free and natural dialogue in multicultural environments and lower language barriers.

[0295] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0296] Step 1:

[0297] The user speaks into the device's microphone. This input audio data is captured by the terminal. The captured audio data undergoes digital signal processing to reduce noise and optimize it.

[0298] Step 2:

[0299] The device sends the captured audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The input is audio data, and the output is text data that reflects its content. This conversion process uses speech signal processing algorithms to analyze and map phonemes.

[0300] Step 3:

[0301] The server receives text data from the terminal and sends it to the translation tool, where it is converted into multiple languages. A generative model (e.g., Google Translate API) receives the input text and translates it into the target language. The data calculations performed here involve natural language processing and string reconstruction based on contextual awareness.

[0302] Step 4:

[0303] The server sends the translated text data to the terminal. The terminal inputs the received translated text into a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The output is the translated speech data. Speech synthesis involves the process of generating a sequence of phonemes from the text and constructing a speech waveform.

[0304] Step 5:

[0305] The device plays the translation results to the user through its speaker. In this step, adjustments are made to ensure that the information is delivered to the user at an appropriate volume and clarity.

[0306] Step 6:

[0307] User reactions and feedback are recorded by the device and sent to the server. The server analyzes the feedback information and uses it to improve the translation algorithm. Here, data mining techniques are used to evaluate translation quality and adjust the algorithm.

[0308] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform specific processing using the user's emotion.

[0309] The present invention provides a system combined with an emotion engine in order to further enhance multilingual communication. This system enables a user to smoothly communicate ideas in different language environments and perform accurate and emotion - considerate communication.

[0310] The user makes a voice input toward the terminal and speaks with emotion. This voice data is captured by the acquisition means of the terminal. The voice data is first converted into text data by the conversion means. Together with the converted text, the voice data is supplied to the emotion engine, and the user's emotional state is analyzed.

[0311] Using the emotion information obtained by this analysis, when the server translates the text data into multiple languages by the translation means, it generates an expression that reflects the user's emotion. The generation model provides a more natural and emotion - conveying translation result by considering the linguistic nuances based on the emotion information.

[0312] The terminal outputs the translation result that takes into account the generated emotion to the user. If this is, for example, a positive emotion, the expression is adjusted to be more friendly, and if it is negative, adjustments such as adding content to follow up politely are made.

[0313] As a specific example, consider the case where the user speaks "Why did this happen?" while having an angry emotion. The terminal sends this to the emotion engine, and the server can add a phrase that empathizes with the emotion, such as "I understand your frustration.", to the translation result of "Why did this happen in such a way?" based on the emotion information indicating anger.

[0314] This system also features a feedback function that analyzes user feedback to further refine its translation algorithms and sentiment recognition capabilities. This allows for continuous improvement of the overall system performance and enables the provision of a better user experience.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The user inputs voice data into the terminal. The terminal receives this voice data using an acquisition method.

[0318] Step 2:

[0319] The terminal converts the received audio data into text data using a conversion mechanism. Speech recognition technology is used in this process.

[0320] Step 3:

[0321] The device sends the converted text data to the server. Simultaneously, the voice data is sent to the emotion engine for sentiment analysis.

[0322] Step 4:

[0323] The server adjusts the translation content using sentiment information obtained from the sentiment engine during the process of translating text data into the target language specified by the translation means. A generative model is utilized to achieve natural translations that reflect emotions.

[0324] Step 5:

[0325] The server sends the translated text data to the user's device.

[0326] Step 6:

[0327] The device outputs the received translated text as either audio or text. Depending on the selection, text-to-speech technology is used to deliver an emotionally sensitive translation to the user.

[0328] (Example 2)

[0329] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0330] In a multilingual environment, achieving natural communication that appropriately reflects user emotions is challenging. Traditional translation systems often fail to consider user emotions, resulting in a lack of nuance and misunderstandings in communication. Furthermore, mechanisms for improving translation quality through user feedback have been insufficient.

[0331] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0332] In this invention, the server includes means for analyzing information, means for translating the information into multiple languages ​​using the analysis results, and means for receiving feedback information and using it to improve the algorithm. This enables natural and accurate translation that reflects the user's emotions and allows for continuous performance improvement based on feedback.

[0333] "Information" refers to audio data obtained from users and the resulting text data.

[0334] "Means of acquisition" refers to devices or systems for collecting user speech as audio data.

[0335] "Means of conversion" refers to the processes and technologies used to convert collected audio data into text data.

[0336] "Means of analysis" refers to technical methods for analyzing the user's emotional state based on the converted text data.

[0337] "Methods for multilingual translation" refer to processes and models for translating text data into different languages ​​based on analysis results.

[0338] "Means of transmission" refers to communication technology used to transmit translated information to the user's device.

[0339] "Means of output" refers to devices or software that display the translation results in audio or text format to communicate them to the user.

[0340] A "generative model" is an algorithm or program that provides translations while taking into account the nuances and emotional information of language.

[0341] "Feedback information" refers to the evaluations and opinions that users provide regarding the translation results.

[0342] "Algorithm improvement" is the process of improving system performance by utilizing the feedback information received.

[0343] This invention is a system that enhances user communication in a multilingual environment. This system provides translations that take user emotions into consideration, enabling smooth and emotionally sensitive communication.

[0344] The user speaks into the device, and the resulting audio is input. Suitable hardware includes the built-in microphone of a smartphone or tablet, or an external microphone device.

[0345] The device uses speech recognition software to convert this speech data into text data. Specific software examples include "Google Cloud Speech-to-Text" and "Microsoft Azure Speech Service."

[0346] The converted text data and original audio are sent to the emotion engine. The emotion engine analyzes the user's emotional state using tools such as "IBM Watson Tone Analyzer".

[0347] The server uses the analyzed sentiment information to translate text data into other languages ​​via a generative AI model. In this process, generative AIs such as "OpenAI GPT-3" and "DeepL" are used to generate expressions that take into account linguistic nuances and hierarchical relationships.

[0348] The translation results are sent to the user's device using the appropriate communication technology. The device then outputs the received translation results to the user via speech synthesis technology or screen display. This ensures that the translation results are conveyed to the user in a natural way that reflects emotional information. For example, if the translation results indicate a negative emotion, a follow-up phrase such as "I understand your frustration" may be added.

[0349] A feedback function is also provided, allowing users to submit opinions and ratings regarding translations. This feedback is used to improve the system's algorithms.

[0350] An example of a prompt is: "Generate an English translation for when the user is angry." Using this prompt, the generative AI model will provide a translation that takes the user's emotions into account.

[0351] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0352] Step 1:

[0353] The user speaks into the device. The user's voice is input and captured by the device's microphone. This input voice contains the user's intentions and emotions. In this step, the voice data is taken into the system.

[0354] Step 2:

[0355] The device uses speech recognition software to convert the input speech data into text data. Here, the speech signal is analyzed and a corresponding string of characters is generated. This conversion is performed using speech recognition technology such as "Google Cloud Speech-to-Text," and the output is text data.

[0356] Step 3:

[0357] The device passes the generated text data and the original audio data to the emotion engine. This engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. This analysis uses tools such as the "IBM Watson Tone Analyzer." The input consists of text data and audio data, and the output is emotion information.

[0358] Step 4:

[0359] The server uses emotional information and text data received from the emotion engine to perform multilingual translation. During this process, generative AI models such as "OpenAI GPT-3" and "DeepL" are used to generate expressions that reflect emotions. This process outputs translated data that incorporates emotional information.

[0360] Step 5:

[0361] The terminal receives translation results sent from the server and outputs them to the user. Output is done through text display or speech synthesis. Specifically, the user is presented with translation results that include expressions that reflect emotions.

[0362] Step 6:

[0363] Users provide feedback on the translation results. This feedback is sent from the terminal to the server as needed and used as data for further system improvements. In this step, feedback information is collected and used to improve the algorithm.

[0364] (Application Example 2)

[0365] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0366] In the multilingual environments we encounter in our daily lives, it's not enough to simply translate languages; we also need to accurately convey the speaker's emotions. However, conventional translation technologies lack the ability to consider emotions, resulting in sluggish communication. Therefore, achieving natural and emotionally rich communication in multiple languages ​​is a challenge.

[0367] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0368] In this invention, the server includes an emotion analysis means for analyzing the user's emotional state, a generation means for generating translation results that reflect the emotional state, and a means for the translation means to consider the nuances of language using a generation model and add expressions based on emotional information. This makes it possible to provide translations that take the user's emotions into consideration and improve the quality of communication.

[0369] "Acquisition means" refers to a device or function for collecting voice data from a user.

[0370] "Conversion means" refers to a device or function for converting acquired audio data into text data.

[0371] "Translation means" refers to a device or function for translating text data into multiple languages.

[0372] "Emotional analysis means" refers to a device or function for analyzing a user's emotional state.

[0373] "Generating means" refers to a device or function for generating translation results that reflect emotional states.

[0374] "Transmission means" refers to a device or function for transferring the generated translation data to the user's terminal.

[0375] "Output means" refers to a device or function for presenting the translation result to the user in audio or text format.

[0376] A "generative model" is an artificial intelligence model used in natural language processing that takes into account the nuances of language and is used for translation and text generation.

[0377] A specific embodiment for carrying out this invention will now be described. The user inputs into the terminal using voice and acquires voice data. The terminal recognizes emotionally charged speech and collects the voice data using an acquisition means. Next, the terminal converts the voice data into text data using a conversion means. This converted text and voice data are sent to a server, where the user's emotional state is analyzed by an emotion analysis means.

[0378] The server uses the analyzed sentiment information to generate translation results that reflect the emotional state through a generation mechanism. The translation mechanism employs a generation AI model, which considers linguistic nuances and sentiment information. Additional expressions are adjusted based on the sentiment information. The generated translation data is transmitted to the terminal via a transmission mechanism and finally presented to the user as audio or text by an output mechanism.

[0379] For example, when the user says "I'm very tired today...", the system recognizes the emotion and adds an emotionally rich expression such as "I hope you can relax and enjoy your meal!" when translating it into English. Also, as a specific example of the prompt sentence, "Please generate a translation of the Japanese sentence '今日はとても疲れていて…' into English, considering the sentiment to be positive and adding an encouraging follow-up. Use the emotional tone to craft a naturally flowing response." can be cited.

[0380] In this way, the system strengthens multilingual emotional communication with the user and realizes natural and smooth conversations.

[0381] The flow of the specific process in Application Example 2 will be described using FIG. 14.

[0382] Step 1:

[0383] The user inputs voice through the microphone. This voice data is acquired by the terminal. The voice input is a natural and concrete utterance and is transmitted to the terminal as voice waveform data.

[0384] Step 2:

[0385] The terminal converts the acquired voice data into text data using a voice recognition engine. The voice waveform data is input, analyzed by the voice recognition engine, and output as text data in string format.

[0386] Step 3:

[0387] The terminal sends the converted text and audio data to the server. The data sent is text data in string format and audio waveform data.

[0388] Step 4:

[0389] The server inputs the received text data into a sentiment analysis device to analyze the user's emotional state. After the text data is analyzed, metadata indicating the user's emotions is output.

[0390] Step 5:

[0391] The server uses emotional metadata to translate text data into multiple languages ​​using a generative AI model and translation tools. The input data consists of text data and emotional metadata, and the output is translated text that takes emotions into account.

[0392] Step 6:

[0393] The server adds emotion-based additional expressions to the translated text through a generation mechanism. The input is the translated text, and the output is the adjusted translated text that reflects the emotion.

[0394] Step 7:

[0395] The server sends the generated translated text to the terminal. This transmitted data is an adjusted translated text that takes sentiment into account.

[0396] Step 8:

[0397] The terminal presents the received translated text to the user using an output device. The user receives the final translation result in either audio or text format.

[0398] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0399] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0400] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0401] [Third Embodiment]

[0402] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0403] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0404] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0405] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0406] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0407] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0408] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0409] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0410] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0411] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0412] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0413] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0414] This invention is a system specifically designed to facilitate communication between people who speak different languages ​​in a multicultural society. The system is designed around multilingual translation and interpretation functions to support user communication in real time.

[0415] The user provides voice input to the device in their own language. This voice data is captured by an acquisition mechanism within the device. Once the voice data is acquired, the device automatically converts it into text data using a conversion mechanism. This process utilizes natural language processing technology.

[0416] The server receives the converted text data and translates it into the specified target language using a translation tool. This translation process utilizes a generative model to achieve natural and fluent translations that are contextually accurate, rather than simply word-for-word matching. Once the translation is complete, the server sends the translated text to the terminal.

[0417] The device uses the received translated text to output its content to the user in either voice or text format. If voice output is requested, the output device within the device converts the text into voice data and conveys the translated content to the user.

[0418] As a concrete example, if a user says "What would you like to eat today?" in Japanese, the device captures this and converts it to text. The server receives this text, translates it into English, and sends the translated result, "What would you like to eat today?", back to the device. The device then plays this English sentence back to the user as audio, enabling smooth communication with someone who speaks a different language.

[0419] Furthermore, the system receives feedback from users, and the server uses this information to improve the translation algorithm through learning mechanisms. This iterative learning process continuously improves the overall translation accuracy of the system.

[0420] In this way, the present invention provides an effective means of enabling free and natural communication among people who speak different languages ​​and reducing barriers to dialogue in multicultural societies.

[0421] The following describes the processing flow.

[0422] Step 1:

[0423] The user provides voice input in their own language to the device. This voice data is captured by the device's built-in acquisition mechanism.

[0424] Step 2:

[0425] The terminal passes the captured audio data to a conversion device, which then converts the audio data into text data. This conversion is performed using speech recognition technology.

[0426] Step 3:

[0427] The converted text data is sent from the terminal to the server. The server then passes the received text data to the translation device.

[0428] Step 4:

[0429] The server translates text data into the specified target language using translation tools. A generative model is used to produce a natural translation that takes linguistic nuances into account.

[0430] Step 5:

[0431] The translated text data is sent from the server to the terminal. The terminal then prepares the data for easy viewing and guidance.

[0432] Step 6:

[0433] The device outputs the received translated text to the user. If voice output is selected, it uses text-to-speech technology to output the translation result to the user.

[0434] (Example 1)

[0435] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0436] In multicultural societies, language barriers exist in communication between speakers of different languages. This makes real-time communication difficult, and there is a need for means to facilitate smooth communication based on mutual understanding of each other's cultural backgrounds and linguistic characteristics.

[0437] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0438] In this invention, the server includes a device for acquiring acoustic information from a user, a device for converting the acquired acoustic information into symbolic information, a device for translating the symbolic information into another language, a device for transmitting the translated symbolic information to the user's information terminal, and a device for outputting the translation result to the user in sound or symbols. This makes it possible to provide natural and fluent translations that take into account context and cultural nuances between users of different languages, and to support real-time communication.

[0439] "Acoustic information" refers to data recorded by mechanical or electronic means from sound transmitted as vibrations in the air.

[0440] An "information terminal" is an electronic device that processes voice and symbolic data and provides information to the user.

[0441] "Symbolic information" refers to string data that represents language or content according to specific rules.

[0442] A "generative algorithm" is a computational procedure or model for generating new data from existing data.

[0443] "Evaluation information" refers to reactions and opinions collected from users regarding specific operations or functions.

[0444] A "device that outputs sound or symbols" is a device that presents information to the user in audio or visual form.

[0445] A "device for translating into other languages" is a device that has the function of converting information expressed in one language into another language.

[0446] This invention is a system that facilitates communication between users who speak different languages ​​in a multicultural society. The system supports natural and fluent communication by translating the user's speech into other languages ​​in real time.

[0447] When a user speaks a different language, they input audio information into the device in their native language. The device is equipped with a microphone to capture sound, thereby acquiring the audio information. The devices used are typical smartphones and tablets, and by installing speech recognition software (e.g., commercial speech recognition applications) on them, the speech is converted into symbolic information.

[0448] The server receives symbolic information sent from the terminal and uses a generation algorithm to translate it into other languages, taking into account the context and nuances of the language. This translation utilizes a multilingual translation API or a generative AI model (e.g., a general-purpose generative AI platform). The generative AI model learns from user feedback to improve translation accuracy.

[0449] The terminal receives translated symbolic information sent from the server and presents the information according to the output format (audio or symbol) selected by the user. Audio output utilizes speech synthesis software, while symbolic output utilizes screen display functionality. Depending on the settings, the user may experience either audio playback or text display.

[0450] As a concrete example, if user A speaks in Japanese, "What would you like to eat today?", the terminal captures this audio and converts it into symbolic information. The server receives this information and generates the text "What would you like to eat today?" translated into English. The generated text is sent to the terminal, which then outputs this text as speech to user B in English.

[0451] Examples of prompt statements include the following:

[0452] "Translate the Japanese text input by User A via voice input into English, a language that User B understands."

[0453] "To improve translation accuracy, please use user feedback to adjust the parameters of the generative AI model."

[0454] In this way, the system enables communication that takes cultural backgrounds and contexts into account between people who speak different languages.

[0455] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0456] Step 1:

[0457] The user speaks into the device in their own language, inputting audio data as sound information. The device captures this audio using its built-in microphone and temporarily stores it as a digital audio file. At this stage, noise reduction and audio clearing are performed to ensure high-quality input data.

[0458] Step 2:

[0459] The device converts acquired acoustic information into symbolic information (text data) using speech recognition software. The main processing includes phoneme analysis through digital signal processing and word recognition using a language model. The output is the converted audio text, ready for transmission to the cloud server.

[0460] Step 3:

[0461] The terminal sends the converted text data to the server. An efficient protocol is used for transmission to minimize network latency. The server receives this text information and formats the data so that it can be input into the multilingual translation API.

[0462] Step 4:

[0463] The server translates the received symbolic information into the target language using a generation AI model. This process utilizes contextual analysis and natural language processing to understand linguistic nuances and generate appropriate translation results. The output is translated text data, which is then prepared for return to the terminal.

[0464] Step 5:

[0465] The terminal receives the translation results sent from the server. The received text is output to the user in the specified format (sound or symbols). In the case of sound output, speech synthesis software is used to convert the text back into speech and play it through the speaker. If output as symbols, the text is displayed on the screen.

[0466] Step 6:

[0467] Users provide feedback on the system's translation results. They input evaluation information on their device and send it to the server. The server analyzes this feedback and uses it as new training data to retrain the AI ​​model, thereby improving the translation algorithm. Through this learning process, the system's translation accuracy continuously improves.

[0468] (Application Example 1)

[0469] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0470] This addresses the challenge of facilitating smooth communication in home environments where people speak different languages. In multicultural societies, technologies that support dialogue between people who speak different languages ​​are needed to improve the quality of life.

[0471] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0472] In this invention, the server includes an acquisition means for acquiring voice information from a user, a conversion means for converting the acquired voice information into text information, a translation means for converting the text information into multiple languages, a transmission means for delivering the translated text information to the user's device, an output means for presenting the translation results to the user in voice or text, a feedback means for collecting user responses and improving translation accuracy, and a dialogue means adapted to smart devices and supporting intercultural communication in a home environment. This enables smooth intercultural dialogue.

[0473] "Acquisition means" refers to a device or function for acquiring voice information from a user.

[0474] "Conversion means" refers to a function that converts acquired audio information into text information.

[0475] "Translation means" refers to technologies or algorithms for converting text information into multiple languages.

[0476] "Transmission means" refers to the technology for delivering translated text information to the user's device.

[0477] "Output method" refers to a function that presents the translation result to the user in audio or text format.

[0478] A "feedback mechanism" is a technology used to collect user feedback and improve translation accuracy.

[0479] "Dialogue tools" refer to functions adapted to smart devices that support intercultural exchange in the home environment.

[0480] This invention realizes a system using a smart device that enables individual users who speak different languages ​​to communicate smoothly. The system mainly consists of a voice acquisition means, a conversion means, a translation means, a transmission means, an output means, a feedback means, and a dialogue means.

[0481] The server uses the microphone in the device to obtain voice information from the user. This voice information is converted into text information using a speech recognition engine installed on the server or terminal (e.g., Google Cloud Speech-to-Text). The converted text is then translated into multiple target languages ​​by a translation tool. The translation uses a generative model that utilizes natural language processing (e.g., Google Translate API) to provide contextually natural translations.

[0482] The translated text information is sent from the server to the user's terminal and can be output in audio format using a speech synthesis engine (e.g., Amazon Polly). During this process, user responses are collected as feedback, which the server analyzes to improve the accuracy of the translation algorithm.

[0483] For example, when offering tea to a foreign guest at home, if the user says "Would you like some tea?" in Japanese, the system will translate and output this in real time as "Would you like some tea?" in English, enabling smooth communication.

[0484] An example of a prompt using a generative AI model is, "Explain how smart devices can improve dialogue between different languages." In this way, the system provides a means to facilitate free and natural dialogue in multicultural environments and lower language barriers.

[0485] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0486] Step 1:

[0487] The user speaks into the device's microphone. This input audio data is captured by the terminal. The captured audio data undergoes digital signal processing to reduce noise and optimize it.

[0488] Step 2:

[0489] The device sends the captured audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The input is audio data, and the output is text data that reflects its content. This conversion process uses speech signal processing algorithms to analyze and map phonemes.

[0490] Step 3:

[0491] The server receives text data from the terminal and sends it to the translation tool, where it is converted into multiple languages. A generative model (e.g., Google Translate API) receives the input text and translates it into the target language. The data calculations performed here involve natural language processing and string reconstruction based on contextual awareness.

[0492] Step 4:

[0493] The server sends the translated text data to the terminal. The terminal inputs the received translated text into a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The output is the translated speech data. Speech synthesis involves the process of generating a sequence of phonemes from the text and constructing a speech waveform.

[0494] Step 5:

[0495] The device plays the translation results to the user through its speaker. In this step, adjustments are made to ensure that the information is delivered to the user at an appropriate volume and clarity.

[0496] Step 6:

[0497] User reactions and feedback are recorded by the device and sent to the server. The server analyzes the feedback information and uses it to improve the translation algorithm. Here, data mining techniques are used to evaluate translation quality and adjust the algorithm.

[0498] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0499] This invention provides a system that combines an emotion engine to further enhance multilingual communication. This system enables users to communicate smoothly in different language environments and to engage in accurate and emotionally sensitive communication.

[0500] The user makes voice input into the device and speaks with emotion. This voice data is captured by the device's acquisition mechanism. The voice data is first converted into text data by a conversion mechanism. Along with the converted text, the voice data is supplied to an emotion engine, which analyzes the user's emotional state.

[0501] The server uses the sentiment information obtained through this analysis to generate expressions that reflect the user's emotions when translating text data into multiple languages ​​using a translation tool. The generative model provides more natural and emotionally conveying translation results by considering the linguistic nuances based on that sentiment information.

[0502] The device then outputs a translation to the user that takes these generated emotions into consideration. For example, if the emotion is positive, the expression is adjusted to be more approachable; if it is negative, adjustments are made such as adding carefully follow-up content.

[0503] As a concrete example, consider a scenario where a user expresses anger by saying, "Why did this happen?" The device sends this to the emotion engine, and the server, based on the emotional information indicating anger, can translate it as "Why did this happen in such a way?" and add empathetic phrases such as "I understand your frustration."

[0504] This system also features a feedback function that analyzes user feedback to further refine its translation algorithms and sentiment recognition capabilities. This allows for continuous improvement of the overall system performance and enables the provision of a better user experience.

[0505] The following describes the processing flow.

[0506] Step 1:

[0507] The user inputs voice data into the terminal. The terminal receives this voice data using an acquisition method.

[0508] Step 2:

[0509] The terminal converts the received audio data into text data using a conversion mechanism. Speech recognition technology is used in this process.

[0510] Step 3:

[0511] The device sends the converted text data to the server. Simultaneously, the voice data is sent to the emotion engine for sentiment analysis.

[0512] Step 4:

[0513] The server adjusts the translation content using sentiment information obtained from the sentiment engine during the process of translating text data into the target language specified by the translation means. A generative model is utilized to achieve natural translations that reflect emotions.

[0514] Step 5:

[0515] The server sends the translated text data to the user's device.

[0516] Step 6:

[0517] The device outputs the received translated text as either audio or text. Depending on the selection, text-to-speech technology is used to deliver an emotionally sensitive translation to the user.

[0518] (Example 2)

[0519] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0520] In a multilingual environment, achieving natural communication that appropriately reflects user emotions is challenging. Traditional translation systems often fail to consider user emotions, resulting in a lack of nuance and misunderstandings in communication. Furthermore, mechanisms for improving translation quality through user feedback have been insufficient.

[0521] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0522] In this invention, the server includes means for analyzing information, means for translating the information into multiple languages ​​using the analysis results, and means for receiving feedback information and using it to improve the algorithm. This enables natural and accurate translation that reflects the user's emotions and allows for continuous performance improvement based on feedback.

[0523] "Information" refers to audio data obtained from users and the resulting text data.

[0524] "Means of acquisition" refers to devices or systems for collecting user speech as audio data.

[0525] "Means of conversion" refers to the processes and technologies used to convert collected audio data into text data.

[0526] "Means of analysis" refers to technical methods for analyzing the user's emotional state based on the converted text data.

[0527] "Methods for multilingual translation" refer to processes and models for translating text data into different languages ​​based on analysis results.

[0528] "Means of transmission" refers to communication technology used to transmit translated information to the user's device.

[0529] "Means of output" refers to devices or software that display the translation results in audio or text format to communicate them to the user.

[0530] A "generative model" is an algorithm or program that provides translations while taking into account the nuances and emotional information of language.

[0531] "Feedback information" refers to the evaluations and opinions that users provide regarding the translation results.

[0532] "Algorithm improvement" is the process of improving system performance by utilizing the feedback information received.

[0533] This invention is a system that enhances user communication in a multilingual environment. This system provides translations that take user emotions into consideration, enabling smooth and emotionally sensitive communication.

[0534] The user speaks into the device, and the resulting audio is input. Suitable hardware includes the built-in microphone of a smartphone or tablet, or an external microphone device.

[0535] The device uses speech recognition software to convert this speech data into text data. Specific software examples include "Google Cloud Speech-to-Text" and "Microsoft Azure Speech Service."

[0536] The converted text data and original audio are sent to the emotion engine. The emotion engine analyzes the user's emotional state using tools such as "IBM Watson Tone Analyzer".

[0537] The server uses the analyzed sentiment information to translate text data into other languages ​​via a generative AI model. In this process, generative AIs such as "OpenAI GPT-3" and "DeepL" are used to generate expressions that take into account linguistic nuances and hierarchical relationships.

[0538] The translation results are sent to the user's device using the appropriate communication technology. The device then outputs the received translation results to the user via speech synthesis technology or screen display. This ensures that the translation results are conveyed to the user in a natural way that reflects emotional information. For example, if the translation results indicate a negative emotion, a follow-up phrase such as "I understand your frustration" may be added.

[0539] A feedback function is also provided, allowing users to submit opinions and ratings regarding translations. This feedback is used to improve the system's algorithms.

[0540] An example of a prompt is: "Generate an English translation for when the user is angry." Using this prompt, the generative AI model will provide a translation that takes the user's emotions into account.

[0541] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0542] Step 1:

[0543] The user speaks into the device. The user's voice is input and captured by the device's microphone. This input voice contains the user's intentions and emotions. In this step, the voice data is taken into the system.

[0544] Step 2:

[0545] The device uses speech recognition software to convert the input speech data into text data. Here, the speech signal is analyzed and a corresponding string of characters is generated. This conversion is performed using speech recognition technology such as "Google Cloud Speech-to-Text," and the output is text data.

[0546] Step 3:

[0547] The device passes the generated text data and the original audio data to the emotion engine. This engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. This analysis uses tools such as the "IBM Watson Tone Analyzer." The input consists of text data and audio data, and the output is emotion information.

[0548] Step 4:

[0549] The server uses emotional information and text data received from the emotion engine to perform multilingual translation. During this process, generative AI models such as "OpenAI GPT-3" and "DeepL" are used to generate expressions that reflect emotions. This process outputs translated data that incorporates emotional information.

[0550] Step 5:

[0551] The terminal receives translation results sent from the server and outputs them to the user. Output is done through text display or speech synthesis. Specifically, the user is presented with translation results that include expressions that reflect emotions.

[0552] Step 6:

[0553] Users provide feedback on the translation results. This feedback is sent from the terminal to the server as needed and used as data for further system improvements. In this step, feedback information is collected and used to improve the algorithm.

[0554] (Application Example 2)

[0555] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0556] In the multilingual environments we encounter in our daily lives, it's not enough to simply translate languages; we also need to accurately convey the speaker's emotions. However, conventional translation technologies lack the ability to consider emotions, resulting in sluggish communication. Therefore, achieving natural and emotionally rich communication in multiple languages ​​is a challenge.

[0557] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0558] In this invention, the server includes an emotion analysis means for analyzing the user's emotional state, a generation means for generating translation results that reflect the emotional state, and a means for the translation means to consider the nuances of language using a generation model and add expressions based on emotional information. This makes it possible to provide translations that take the user's emotions into consideration and improve the quality of communication.

[0559] "Acquisition means" refers to a device or function for collecting voice data from a user.

[0560] "Conversion means" refers to a device or function for converting acquired audio data into text data.

[0561] "Translation means" refers to a device or function for translating text data into multiple languages.

[0562] "Emotional analysis means" refers to a device or function for analyzing a user's emotional state.

[0563] "Generating means" refers to a device or function for generating translation results that reflect emotional states.

[0564] "Transmission means" refers to a device or function for transferring the generated translation data to the user's terminal.

[0565] "Output means" refers to a device or function for presenting the translation result to the user in audio or text format.

[0566] A "generative model" is an artificial intelligence model used in natural language processing that takes into account the nuances of language and is used for translation and text generation.

[0567] A specific embodiment for carrying out this invention will now be described. The user inputs into the terminal using voice and acquires voice data. The terminal recognizes emotionally charged speech and collects the voice data using an acquisition means. Next, the terminal converts the voice data into text data using a conversion means. This converted text and voice data are sent to a server, where the user's emotional state is analyzed by an emotion analysis means.

[0568] The server uses the analyzed sentiment information to generate translation results that reflect the emotional state through a generation mechanism. The translation mechanism employs a generation AI model, which considers linguistic nuances and sentiment information. Additional expressions are adjusted based on the sentiment information. The generated translation data is transmitted to the terminal via a transmission mechanism and finally presented to the user as audio or text by an output mechanism.

[0569] For example, when the user says "I'm very tired today...", the system recognizes the emotion and adds an emotionally rich expression such as "I hope you can relax and enjoy your meal!" when translating it into English. Also, as a specific example of the prompt sentence, "Please generate a translation of the Japanese sentence '今日はとても疲れていて…' into English, considering the sentiment to be positive and adding an encouraging follow-up. Use the emotional tone to craft a naturally flowing response." can be cited.

[0570] In this way, the system strengthens multilingual emotional communication with the user and realizes natural and smooth conversations.

[0571] The flow of the specific process in Application Example 2 will be described using FIG. 14.

[0572] Step 1:

[0573] The user inputs voice through the microphone. This voice data is acquired by the terminal. The voice input is a natural and concrete utterance and is transmitted to the terminal as voice waveform data.

[0574] Step 2:

[0575] The terminal converts the acquired voice data into text data using a voice recognition engine. The voice waveform data is input, analyzed by the voice recognition engine, and output as text data in string format.

[0576] Step 3:

[0577] The terminal sends the converted text and audio data to the server. The data sent is text data in string format and audio waveform data.

[0578] Step 4:

[0579] The server inputs the received text data into a sentiment analysis device to analyze the user's emotional state. After the text data is analyzed, metadata indicating the user's emotions is output.

[0580] Step 5:

[0581] The server uses emotional metadata to translate text data into multiple languages ​​using a generative AI model and translation tools. The input data consists of text data and emotional metadata, and the output is translated text that takes emotions into account.

[0582] Step 6:

[0583] The server adds emotion-based additional expressions to the translated text through a generation mechanism. The input is the translated text, and the output is the adjusted translated text that reflects the emotion.

[0584] Step 7:

[0585] The server sends the generated translated text to the terminal. This transmitted data is an adjusted translated text that takes sentiment into account.

[0586] Step 8:

[0587] The terminal presents the received translated text to the user using an output device. The user receives the final translation result in either audio or text format.

[0588] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0589] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0590] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0591] [Fourth Embodiment]

[0592] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0593] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0594] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0595] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0596] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0597] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0598] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0599] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0600] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0601] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0602] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0603] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0604] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0605] This invention is a system specifically designed to facilitate communication between people who speak different languages ​​in a multicultural society. The system is designed around multilingual translation and interpretation functions to support user communication in real time.

[0606] The user provides voice input to the device in their own language. This voice data is captured by an acquisition mechanism within the device. Once the voice data is acquired, the device automatically converts it into text data using a conversion mechanism. This process utilizes natural language processing technology.

[0607] The server receives the converted text data and translates it into the specified target language using a translation tool. This translation process utilizes a generative model to achieve natural and fluent translations that are contextually accurate, rather than simply word-for-word matching. Once the translation is complete, the server sends the translated text to the terminal.

[0608] The device uses the received translated text to output its content to the user in either voice or text format. If voice output is requested, the output device within the device converts the text into voice data and conveys the translated content to the user.

[0609] As a concrete example, if a user says "What would you like to eat today?" in Japanese, the device captures this and converts it to text. The server receives this text, translates it into English, and sends the translated result, "What would you like to eat today?", back to the device. The device then plays this English sentence back to the user as audio, enabling smooth communication with someone who speaks a different language.

[0610] Furthermore, the system receives feedback from users, and the server uses this information to improve the translation algorithm through learning mechanisms. This iterative learning process continuously improves the overall translation accuracy of the system.

[0611] In this way, the present invention provides an effective means of enabling free and natural communication among people who speak different languages ​​and reducing barriers to dialogue in multicultural societies.

[0612] The following describes the processing flow.

[0613] Step 1:

[0614] The user provides voice input in their own language to the device. This voice data is captured by the device's built-in acquisition mechanism.

[0615] Step 2:

[0616] The terminal passes the captured audio data to a conversion device, which then converts the audio data into text data. This conversion is performed using speech recognition technology.

[0617] Step 3:

[0618] The converted text data is sent from the terminal to the server. The server then passes the received text data to the translation device.

[0619] Step 4:

[0620] The server translates text data into the specified target language using translation tools. A generative model is used to produce a natural translation that takes linguistic nuances into account.

[0621] Step 5:

[0622] The translated text data is sent from the server to the terminal. The terminal then prepares the data for easy viewing and guidance.

[0623] Step 6:

[0624] The device outputs the received translated text to the user. If voice output is selected, it uses text-to-speech technology to output the translation result to the user.

[0625] (Example 1)

[0626] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0627] In multicultural societies, language barriers exist in communication between speakers of different languages. This makes real-time communication difficult, and there is a need for means to facilitate smooth communication based on mutual understanding of each other's cultural backgrounds and linguistic characteristics.

[0628] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0629] In this invention, the server includes a device for acquiring acoustic information from a user, a device for converting the acquired acoustic information into symbolic information, a device for translating the symbolic information into another language, a device for transmitting the translated symbolic information to the user's information terminal, and a device for outputting the translation result to the user in sound or symbols. This makes it possible to provide natural and fluent translations that take into account context and cultural nuances between users of different languages, and to support real-time communication.

[0630] "Acoustic information" refers to data recorded by mechanical or electronic means from sound transmitted as vibrations in the air.

[0631] An "information terminal" is an electronic device that processes voice and symbolic data and provides information to the user.

[0632] "Symbolic information" refers to string data that represents language or content according to specific rules.

[0633] A "generative algorithm" is a computational procedure or model for generating new data from existing data.

[0634] "Evaluation information" refers to reactions and opinions collected from users regarding specific operations or functions.

[0635] A "device that outputs sound or symbols" is a device that presents information to the user in audio or visual form.

[0636] A "device for translating into other languages" is a device that has the function of converting information expressed in one language into another language.

[0637] This invention is a system that facilitates communication between users who speak different languages ​​in a multicultural society. The system supports natural and fluent communication by translating the user's speech into other languages ​​in real time.

[0638] When a user speaks a different language, they input audio information into the device in their native language. The device is equipped with a microphone to capture sound, thereby acquiring the audio information. The devices used are typical smartphones and tablets, and by installing speech recognition software (e.g., commercial speech recognition applications) on them, the speech is converted into symbolic information.

[0639] The server receives symbolic information sent from the terminal and uses a generation algorithm to translate it into other languages, taking into account the context and nuances of the language. This translation utilizes a multilingual translation API or a generative AI model (e.g., a general-purpose generative AI platform). The generative AI model learns from user feedback to improve translation accuracy.

[0640] The terminal receives translated symbolic information sent from the server and presents the information according to the output format (audio or symbol) selected by the user. Audio output utilizes speech synthesis software, while symbolic output utilizes screen display functionality. Depending on the settings, the user may experience either audio playback or text display.

[0641] As a concrete example, if user A speaks in Japanese, "What would you like to eat today?", the terminal captures this audio and converts it into symbolic information. The server receives this information and generates the text "What would you like to eat today?" translated into English. The generated text is sent to the terminal, which then outputs this text as speech to user B in English.

[0642] Examples of prompt statements include the following:

[0643] "Translate the Japanese text input by User A via voice input into English, a language that User B understands."

[0644] "To improve translation accuracy, please use user feedback to adjust the parameters of the generative AI model."

[0645] In this way, the system enables communication that takes cultural backgrounds and contexts into account between people who speak different languages.

[0646] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0647] Step 1:

[0648] The user speaks into the device in their own language, inputting audio data as sound information. The device captures this audio using its built-in microphone and temporarily stores it as a digital audio file. At this stage, noise reduction and audio clearing are performed to ensure high-quality input data.

[0649] Step 2:

[0650] The device converts acquired acoustic information into symbolic information (text data) using speech recognition software. The main processing includes phoneme analysis through digital signal processing and word recognition using a language model. The output is the converted audio text, ready for transmission to the cloud server.

[0651] Step 3:

[0652] The terminal sends the converted text data to the server. An efficient protocol is used for transmission to minimize network latency. The server receives this text information and formats the data so that it can be input into the multilingual translation API.

[0653] Step 4:

[0654] The server translates the received symbolic information into the target language using a generation AI model. This process utilizes contextual analysis and natural language processing to understand linguistic nuances and generate appropriate translation results. The output is translated text data, which is then prepared for return to the terminal.

[0655] Step 5:

[0656] The terminal receives the translation results sent from the server. The received text is output to the user in the specified format (sound or symbols). In the case of sound output, speech synthesis software is used to convert the text back into speech and play it through the speaker. If output as symbols, the text is displayed on the screen.

[0657] Step 6:

[0658] Users provide feedback on the system's translation results. They input evaluation information on their device and send it to the server. The server analyzes this feedback and uses it as new training data to retrain the AI ​​model, thereby improving the translation algorithm. Through this learning process, the system's translation accuracy continuously improves.

[0659] (Application Example 1)

[0660] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0661] This addresses the challenge of facilitating smooth communication in home environments where people speak different languages. In multicultural societies, technologies that support dialogue between people who speak different languages ​​are needed to improve the quality of life.

[0662] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0663] In this invention, the server includes an acquisition means for acquiring voice information from a user, a conversion means for converting the acquired voice information into text information, a translation means for converting the text information into multiple languages, a transmission means for delivering the translated text information to the user's device, an output means for presenting the translation results to the user in voice or text, a feedback means for collecting user responses and improving translation accuracy, and a dialogue means adapted to smart devices and supporting intercultural communication in a home environment. This enables smooth intercultural dialogue.

[0664] "Acquisition means" refers to a device or function for acquiring voice information from a user.

[0665] "Conversion means" refers to a function that converts acquired audio information into text information.

[0666] "Translation means" refers to technologies or algorithms for converting text information into multiple languages.

[0667] "Transmission means" refers to the technology for delivering translated text information to the user's device.

[0668] "Output method" refers to a function that presents the translation result to the user in audio or text format.

[0669] A "feedback mechanism" is a technology used to collect user feedback and improve translation accuracy.

[0670] "Dialogue tools" refer to functions adapted to smart devices that support intercultural exchange in the home environment.

[0671] This invention realizes a system using a smart device that enables individual users who speak different languages ​​to communicate smoothly. The system mainly consists of a voice acquisition means, a conversion means, a translation means, a transmission means, an output means, a feedback means, and a dialogue means.

[0672] The server uses the microphone in the device to obtain voice information from the user. This voice information is converted into text information using a speech recognition engine installed on the server or terminal (e.g., Google Cloud Speech-to-Text). The converted text is then translated into multiple target languages ​​by a translation tool. The translation uses a generative model that utilizes natural language processing (e.g., Google Translate API) to provide contextually natural translations.

[0673] The translated text information is sent from the server to the user's terminal and can be output in audio format using a speech synthesis engine (e.g., Amazon Polly). During this process, user responses are collected as feedback, which the server analyzes to improve the accuracy of the translation algorithm.

[0674] For example, when offering tea to a foreign guest at home, if the user says "Would you like some tea?" in Japanese, the system will translate and output this in real time as "Would you like some tea?" in English, enabling smooth communication.

[0675] An example of a prompt using a generative AI model is, "Explain how smart devices can improve dialogue between different languages." In this way, the system provides a means to facilitate free and natural dialogue in multicultural environments and lower language barriers.

[0676] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0677] Step 1:

[0678] The user speaks into the device's microphone. This input audio data is captured by the terminal. The captured audio data undergoes digital signal processing to reduce noise and optimize it.

[0679] Step 2:

[0680] The device sends the captured audio data to a speech recognition engine (e.g., Google Cloud Speech-to-Text) and converts it into text data. The input is audio data, and the output is text data that reflects its content. This conversion process uses speech signal processing algorithms to analyze and map phonemes.

[0681] Step 3:

[0682] The server receives text data from the terminal and sends it to the translation tool, where it is converted into multiple languages. A generative model (e.g., Google Translate API) receives the input text and translates it into the target language. The data calculations performed here involve natural language processing and string reconstruction based on contextual awareness.

[0683] Step 4:

[0684] The server sends the translated text data to the terminal. The terminal inputs the received translated text into a speech synthesis engine (e.g., Amazon Polly) and converts it into speech data. The output is the translated speech data. Speech synthesis involves the process of generating a sequence of phonemes from the text and constructing a speech waveform.

[0685] Step 5:

[0686] The device plays the translation results to the user through its speaker. In this step, adjustments are made to ensure that the information is delivered to the user at an appropriate volume and clarity.

[0687] Step 6:

[0688] User reactions and feedback are recorded by the device and sent to the server. The server analyzes the feedback information and uses it to improve the translation algorithm. Here, data mining techniques are used to evaluate translation quality and adjust the algorithm.

[0689] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0690] This invention provides a system that combines an emotion engine to further enhance multilingual communication. This system enables users to communicate smoothly in different language environments and to engage in accurate and emotionally sensitive communication.

[0691] The user makes voice input into the device and speaks with emotion. This voice data is captured by the device's acquisition mechanism. The voice data is first converted into text data by a conversion mechanism. Along with the converted text, the voice data is supplied to an emotion engine, which analyzes the user's emotional state.

[0692] The server uses the sentiment information obtained through this analysis to generate expressions that reflect the user's emotions when translating text data into multiple languages ​​using a translation tool. The generative model provides more natural and emotionally conveying translation results by considering the linguistic nuances based on that sentiment information.

[0693] The device then outputs a translation to the user that takes these generated emotions into consideration. For example, if the emotion is positive, the expression is adjusted to be more approachable; if it is negative, adjustments are made such as adding carefully follow-up content.

[0694] As a concrete example, consider a scenario where a user expresses anger by saying, "Why did this happen?" The device sends this to the emotion engine, and the server, based on the emotional information indicating anger, can translate it as "Why did this happen in such a way?" and add empathetic phrases such as "I understand your frustration."

[0695] This system also features a feedback function that analyzes user feedback to further refine its translation algorithms and sentiment recognition capabilities. This allows for continuous improvement of the overall system performance and enables the provision of a better user experience.

[0696] The following describes the processing flow.

[0697] Step 1:

[0698] The user inputs voice data into the terminal. The terminal receives this voice data using an acquisition method.

[0699] Step 2:

[0700] The terminal converts the received audio data into text data using a conversion mechanism. Speech recognition technology is used in this process.

[0701] Step 3:

[0702] The device sends the converted text data to the server. Simultaneously, the voice data is sent to the emotion engine for sentiment analysis.

[0703] Step 4:

[0704] The server adjusts the translation content using sentiment information obtained from the sentiment engine during the process of translating text data into the target language specified by the translation means. A generative model is utilized to achieve natural translations that reflect emotions.

[0705] Step 5:

[0706] The server sends the translated text data to the user's device.

[0707] Step 6:

[0708] The device outputs the received translated text as either audio or text. Depending on the selection, text-to-speech technology is used to deliver an emotionally sensitive translation to the user.

[0709] (Example 2)

[0710] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0711] In a multilingual environment, achieving natural communication that appropriately reflects user emotions is challenging. Traditional translation systems often fail to consider user emotions, resulting in a lack of nuance and misunderstandings in communication. Furthermore, mechanisms for improving translation quality through user feedback have been insufficient.

[0712] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0713] In this invention, the server includes means for analyzing information, means for translating the information into multiple languages ​​using the analysis results, and means for receiving feedback information and using it to improve the algorithm. This enables natural and accurate translation that reflects the user's emotions and allows for continuous performance improvement based on feedback.

[0714] "Information" refers to audio data obtained from users and the resulting text data.

[0715] "Means of acquisition" refers to devices or systems for collecting user speech as audio data.

[0716] "Means of conversion" refers to the processes and technologies used to convert collected audio data into text data.

[0717] "Means of analysis" refers to technical methods for analyzing the user's emotional state based on the converted text data.

[0718] "Methods for multilingual translation" refer to processes and models for translating text data into different languages ​​based on analysis results.

[0719] "Means of transmission" refers to communication technology used to transmit translated information to the user's device.

[0720] "Means of output" refers to devices or software that display the translation results in audio or text format to communicate them to the user.

[0721] A "generative model" is an algorithm or program that provides translations while taking into account the nuances and emotional information of language.

[0722] "Feedback information" refers to the evaluations and opinions that users provide regarding the translation results.

[0723] "Algorithm improvement" is the process of improving system performance by utilizing the feedback information received.

[0724] This invention is a system that enhances user communication in a multilingual environment. This system provides translations that take user emotions into consideration, enabling smooth and emotionally sensitive communication.

[0725] The user speaks into the device, and the resulting audio is input. Suitable hardware includes the built-in microphone of a smartphone or tablet, or an external microphone device.

[0726] The device uses speech recognition software to convert this speech data into text data. Specific software examples include "Google Cloud Speech-to-Text" and "Microsoft Azure Speech Service."

[0727] The converted text data and original audio are sent to the emotion engine. The emotion engine analyzes the user's emotional state using tools such as "IBM Watson Tone Analyzer".

[0728] The server uses the analyzed sentiment information to translate text data into other languages ​​via a generative AI model. In this process, generative AIs such as "OpenAI GPT-3" and "DeepL" are used to generate expressions that take into account linguistic nuances and hierarchical relationships.

[0729] The translation results are sent to the user's device using the appropriate communication technology. The device then outputs the received translation results to the user via speech synthesis technology or screen display. This ensures that the translation results are conveyed to the user in a natural way that reflects emotional information. For example, if the translation results indicate a negative emotion, a follow-up phrase such as "I understand your frustration" may be added.

[0730] A feedback function is also provided, allowing users to submit opinions and ratings regarding translations. This feedback is used to improve the system's algorithms.

[0731] An example of a prompt is: "Generate an English translation for when the user is angry." Using this prompt, the generative AI model will provide a translation that takes the user's emotions into account.

[0732] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0733] Step 1:

[0734] The user speaks into the device. The user's voice is input and captured by the device's microphone. This input voice contains the user's intentions and emotions. In this step, the voice data is taken into the system.

[0735] Step 2:

[0736] The device uses speech recognition software to convert the input speech data into text data. Here, the speech signal is analyzed and a corresponding string of characters is generated. This conversion is performed using speech recognition technology such as "Google Cloud Speech-to-Text," and the output is text data.

[0737] Step 3:

[0738] The device passes the generated text data and the original audio data to the emotion engine. This engine analyzes the tone of the voice and the content of the text to identify the user's emotional state. This analysis uses tools such as the "IBM Watson Tone Analyzer." The input consists of text data and audio data, and the output is emotion information.

[0739] Step 4:

[0740] The server uses emotional information and text data received from the emotion engine to perform multilingual translation. During this process, generative AI models such as "OpenAI GPT-3" and "DeepL" are used to generate expressions that reflect emotions. This process outputs translated data that incorporates emotional information.

[0741] Step 5:

[0742] The terminal receives translation results sent from the server and outputs them to the user. Output is done through text display or speech synthesis. Specifically, the user is presented with translation results that include expressions that reflect emotions.

[0743] Step 6:

[0744] Users provide feedback on the translation results. This feedback is sent from the terminal to the server as needed and used as data for further system improvements. In this step, feedback information is collected and used to improve the algorithm.

[0745] (Application Example 2)

[0746] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0747] In the multilingual environments we encounter in our daily lives, it's not enough to simply translate languages; we also need to accurately convey the speaker's emotions. However, conventional translation technologies lack the ability to consider emotions, resulting in sluggish communication. Therefore, achieving natural and emotionally rich communication in multiple languages ​​is a challenge.

[0748] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0749] In this invention, the server includes an emotion analysis means for analyzing the user's emotional state, a generation means for generating translation results that reflect the emotional state, and a means for the translation means to consider the nuances of language using a generation model and add expressions based on emotional information. This makes it possible to provide translations that take the user's emotions into consideration and improve the quality of communication.

[0750] "Acquisition means" refers to a device or function for collecting voice data from a user.

[0751] "Conversion means" refers to a device or function for converting acquired audio data into text data.

[0752] "Translation means" refers to a device or function for translating text data into multiple languages.

[0753] "Emotional analysis means" refers to a device or function for analyzing a user's emotional state.

[0754] "Generating means" refers to a device or function for generating translation results that reflect emotional states.

[0755] "Transmission means" refers to a device or function for transferring the generated translation data to the user's terminal.

[0756] "Output means" refers to a device or function for presenting the translation result to the user in audio or text format.

[0757] A "generative model" is an artificial intelligence model used for translation and text generation in natural language processing, taking into account the nuances of language.

[0758] A specific embodiment for implementing this invention will be described. The user inputs to the terminal using voice to obtain voice data. The terminal recognizes the utterance with emotion and collects the voice data by an acquisition means. Next, the terminal converts the voice data into text data using a conversion means. This converted text and voice data are transmitted to a server, and the emotional state of the user is analyzed by an emotion analysis means.

[0759] The server uses the analyzed emotion information to generate a translation result reflecting the emotional state through a generation means. A generative AI model is used for the translation means, and a translation considering the nuances of language and emotion information is performed. At this time, additional expressions are adjusted based on the emotion information. The generated translation data is transmitted to the terminal via a transmission means and finally presented to the user in voice or text by an output means.

[0760] As an example, when the user says "I'm very tired today...", the system recognizes the emotion and adds an emotionally rich expression such as "I hope you can relax and enjoy your meal!" when translating it into English. Also, as a specific example of a prompt sentence, "Please generate a translation of the Japanese sentence '今日はとても疲れていて…' into English, considering the sentiment to be positive and adding an encouraging follow-up. Use the emotional tone to craft a naturally flowing response." can be cited.

[0761] In this way, the system enhances multilingual emotional communication with users, enabling natural and smooth dialogue.

[0762] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0763] Step 1:

[0764] The user inputs voice through the microphone. This voice data is acquired by the device. The voice input is natural and concrete speech and is transmitted to the device as audio waveform data.

[0765] Step 2:

[0766] The terminal converts the acquired audio data into text data using a speech recognition engine. Audio waveform data is input, analyzed by the speech recognition engine, and output as text data in string format.

[0767] Step 3:

[0768] The terminal sends the converted text and audio data to the server. The data sent is text data in string format and audio waveform data.

[0769] Step 4:

[0770] The server inputs the received text data into a sentiment analysis device to analyze the user's emotional state. After the text data is analyzed, metadata indicating the user's emotions is output.

[0771] Step 5:

[0772] The server uses emotional metadata to translate text data into multiple languages ​​using a generative AI model and translation tools. The input data consists of text data and emotional metadata, and the output is translated text that takes emotions into account.

[0773] Step 6:

[0774] The server adds emotion-based additional expressions to the translated text through a generation mechanism. The input is the translated text, and the output is the adjusted translated text that reflects the emotion.

[0775] Step 7:

[0776] The server sends the generated translated text to the terminal. This transmitted data is an adjusted translated text that takes sentiment into account.

[0777] Step 8:

[0778] The terminal presents the received translated text to the user using an output device. The user receives the final translation result in either audio or text format.

[0779] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0780] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0781] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0782] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0783] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0784] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0785] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0786] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0787] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0788] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0789] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0790] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0791] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0792] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0793] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0794] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0795] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0796] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0797] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0798] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0799] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0800] The following is further disclosed regarding the embodiments described above.

[0801] (Claim 1)

[0802] A means of acquiring voice data from the user,

[0803] A conversion means for converting acquired audio data into text data,

[0804] Translation methods for translating text data into multiple languages,

[0805] A transmission means for sending translated text data to the user's terminal,

[0806] An output means for outputting the translation result to the user in audio or text,

[0807] A system that includes this.

[0808] (Claim 2)

[0809] The system according to claim 1, wherein the translation means provides a translation that takes into account the nuances of language using a generative model.

[0810] (Claim 3)

[0811] The system according to claim 1, further comprising a learning means for receiving user feedback information and using it to improve the translation algorithm.

[0812] "Example 1"

[0813] (Claim 1)

[0814] A device that acquires acoustic information from the user,

[0815] A device that converts acquired acoustic information into symbolic information,

[0816] A device for translating symbolic information into other languages,

[0817] A device that transmits translated symbolic information to the user's information terminal,

[0818] A device that outputs the translation result to the user in the form of sound or symbols,

[0819] A system that includes this.

[0820] (Claim 2)

[0821] The system according to claim 1, wherein the translation device provides a translation that takes into account the details of the language using a generation algorithm.

[0822] (Claim 3)

[0823] The system according to claim 1, further comprising a learning device that receives user evaluation information and uses it to improve the translation algorithm.

[0824] "Application Example 1"

[0825] (Claim 1)

[0826] A means of obtaining voice information from the user,

[0827] A conversion means for converting acquired audio information into text information,

[0828] A translation method that converts text information into multiple languages,

[0829] A transmission means for delivering translated text information to the user's device,

[0830] An output means for presenting the translation result to the user in audio or text,

[0831] A feedback mechanism to collect user feedback and improve translation accuracy,

[0832] A system that includes this.

[0833] (Claim 2)

[0834] The system according to claim 1, wherein the translation means provides a translation that takes into account the nuances of language using a generation algorithm.

[0835] (Claim 3)

[0836] The system according to claim 1, further comprising a means for dialogue that is adapted to smart devices and supports intercultural exchange in a home environment.

[0837] "Example 2 of combining an emotion engine"

[0838] (Claim 1)

[0839] Means of obtaining information,

[0840] A means of transforming the acquired information,

[0841] A means of analyzing the converted information,

[0842] A means of translating information into multiple languages ​​using the analysis results,

[0843] A means of sending translated information to a terminal,

[0844] A means of outputting the translation result,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, which provides translations that take emotional information into account using a generative model.

[0848] (Claim 3)

[0849] The system according to claim 1, further comprising means for receiving feedback information and using it to improve the algorithm.

[0850] "Application example 2 when combining with an emotional engine"

[0851] (Claim 1)

[0852] A means of acquiring voice data from the user,

[0853] A conversion means for converting acquired audio data into text data,

[0854] Translation methods for translating text data into multiple languages,

[0855] A sentiment analysis tool for analyzing the emotional state of users,

[0856] A generation means for generating translation results that reflect emotional states,

[0857] A transmission means for sending translated text data to the user's terminal,

[0858] An output means for outputting the translation result to the user in audio or text,

[0859] A system that includes this.

[0860] (Claim 2)

[0861] The system according to claim 1, wherein the translation means uses a generative model to consider the nuances of language and adds expressions based on emotional information.

[0862] (Claim 3)

[0863] The system according to claim 1, further comprising a learning means for receiving user feedback information and using it to improve the translation algorithm and sentiment recognition function. [Explanation of symbols]

[0864] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring voice data from the user, A conversion means for converting acquired audio data into text data, Translation methods for translating text data into multiple languages, A transmission means for sending translated text data to the user's terminal, An output means for outputting the translation result to the user in audio or text, A system that includes this.

2. The system according to claim 1, wherein the translation means provides a translation that takes into account the nuances of language using a generative model.

3. The system according to claim 1, further comprising a learning means for receiving user feedback information and using it to improve the translation algorithm.

Images