system

Abstract

We provide the system. [Solution] Means of collecting voice from users, A means of analyzing collected audio data to identify language and intonation, Based on the analysis results, a means of estimating the cultural background using a generative model, A means of adjusting cross-cultural communication according to the estimated cultural background and translating and interpreting the user's statements, A means to detect the risk of cultural taboos and misunderstandings and warn users, Means to support dialogue between individuals with different cultural backgrounds within the family, A system that includes this.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional translation systems mainly specialize in language conversion in a simple context and cannot take into account differences in the cultural backgrounds and values of interlocutors, which may result in misunderstandings and cultural conflicts. In such systems, it is difficult to support smooth communication between different cultures. To solve this problem, there is a need for a technology that not only performs language conversion but also grasps the cultural background of the interlocutors and leads to appropriate expressions accordingly.

Means for Solving the Problems

[0005] This invention utilizes a device to collect user voice data, analyzes the collected voice data to identify language and intonation, and then uses a generative model to estimate the cultural background based on the results. This process enables the system to translate and interpret the user's statements according to the estimated cultural background. Furthermore, it includes a device to detect cultural taboos and the risk of misunderstanding, and issues voice warnings to the user to facilitate smooth communication. In addition, this system analyzes the other party's reactions in real time and provides feedback on changes to the user, thereby providing advanced interpretation based on cultural considerations.

[0006] A "user" refers to a person who uses this translation system to engage in dialogue or communication.

[0007] "Audio data" refers to digital information that is electronically collected and recorded from the voices of the user or the other party.

[0008] "Device" refers to a part of a machine or software used to collect, analyze, and convert audio.

[0009] "Analysis" refers to the process of identifying necessary information from collected audio data and determining language and intonation.

[0010] "Language" refers to a system of spoken or written language used for communication between different cultural spheres.

[0011] "Intonation" refers to changes in pitch and stress in speech, and is an element that can sometimes reveal the speaker's regional and emotional background.

[0012] A "generative model" is a computational model within a system that uses machine learning or other advanced technologies to make inferences or generate information from input data.

[0013] "Cultural background" refers to the social framework that includes the cultural characteristics, values, and customs of an individual or group.

[0014] "Interpretation" refers to the act or technique of appropriately translating content expressed in one language into another language for transmission.

[0015] A "cultural taboo" refers to an action, topic, or expression that is considered to be avoided or inappropriate in a particular culture.

[0016] "Risk of misunderstanding" refers to the possibility of a mismatch in intent or misinterpretation between the speaker and the listener.

[0017] "Voice warning" refers to an alert that is issued in voice to inform the user of dangers or points of caution. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9]Shows an emotion map to which a plurality of emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention includes a series of devices and processes necessary to implement an interpretation system that takes cultural background into account. A device worn by the user, such as an earphone or smartphone, collects audio data. This allows the conversation between the user and the other party to be monitored in real time and transmitted to a server.

[0040] The server performs speech recognition and analysis on the received audio data to identify the language and intonation of the conversation. Using generative models based on this identified information, it estimates the other party's cultural background. This allows the server to transform the user's statements into appropriate expressions that take into account the other party's communication style and values.

[0041] The server performs a translation process according to the cultural background, sends the converted audio information back to the user's device, and outputs it through the earphones. This allows the user to receive messages in a structure appropriate to the other person's culture.

[0042] Furthermore, the server constantly monitors the context of the conversation and, if it detects a risk of cultural taboos or misunderstandings, sends an alert to the user via voice warning. This alert includes appropriate alternative expressions and suggestions for changing the topic to help the user avoid misunderstandings.

[0043] One concrete example is a conversation between a Japanese-speaking user and a French-speaking partner where a topic related to a specific expression in French culture is discussed, and it is then translated into something that is commonly discussed in Japanese culture. For instance, if a topic of a particular holiday or custom popular in France comes up, it could be changed to a discussion of a similar holiday or custom in Japan, thereby promoting cross-cultural understanding.

[0044] This system will facilitate smoother and more considerate international and intercultural communication, enabling dialogue with minimal misunderstandings.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The device collects the conversation audio between the user and the other party in real time using the microphone in the earphone. The collected audio data is then de-noised and prepared for the next processing step as clear audio data.

[0048] Step 2:

[0049] The terminal sends the pre-processed audio data to the server via a network such as the internet. The network connection utilizes encryption protocols for data protection.

[0050] Step 3:

[0051] The server places the received audio data into a queue for analysis. During this process, speech recognition technology is used to identify the language being used in the audio data.

[0052] Step 4:

[0053] The server analyzes the intonation of the audio data to estimate the speaker's regional or cultural background. It identifies geographical features based on tone, rhythm, and accent.

[0054] Step 5:

[0055] The server uses a generative model to generate cultural profiles based on identified cultural elements, such as politeness and directness.

[0056] Step 6:

[0057] Based on the generated cultural profile, the server translates the user's utterances into language that is easily understood by the recipient. This translation is adjusted to match the selected context.

[0058] Step 7:

[0059] The server sends the converted language data back to the terminal. The terminal converts this data into an audio signal and transmits it to the user in real time via earphones.

[0060] Step 8:

[0061] The server performs continuous response analysis to detect potential cultural taboos and misunderstandings during conversations. If a risk is detected, the server sends an audio warning to the user.

[0062] Step 9:

[0063] The server generates alternative expressions and suggestions for new topics based on the risk of misunderstanding, and provides these to the user along with alerts. This helps the user to appropriately adjust the conversation.

[0064] (Example 1)

[0065] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0066] In today's global society, opportunities for people with different languages ​​and cultural backgrounds to communicate are increasing. In these situations, not only language barriers but also cultural misunderstandings and taboos can become major obstacles. Traditional interpretation systems focus on language conversion, but they often lack sufficient consideration and adaptation to cultural backgrounds, potentially hindering smooth communication and leading to misunderstandings.

[0067] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0068] In this invention, the server includes means for collecting auditory information from the user, means for analyzing the collected auditory information to identify linguistic features and intonation, and means for estimating the cultural background using a generation algorithm based on the analysis results. This enables smooth intercultural communication by providing appropriate interpretation and warnings while considering the cultural context.

[0069] "User" refers to a person who uses the system to collect and interpret audio data.

[0070] "Auditory information" refers to data that includes audio data and language information collected from users.

[0071] "Linguistic features" refer to the types of language, unique expressions, and vocabulary patterns extracted from audio data.

[0072] "Intonation" refers to elements that describe the characteristics of speech in language, such as intonation, volume, and rhythm.

[0073] A "generative algorithm" refers to a set of computational procedures or models used to generate or transform information.

[0074] "Cultural background" refers to the values, beliefs, and norms that are generally shared within a particular culture or society.

[0075] "Interpretation" refers to the act of facilitating mutual understanding by appropriately conveying content expressed in one language into another language.

[0076] A "cultural taboo" refers to actions, expressions, or topics that are considered to be avoided in a particular culture.

[0077] The "risk of misunderstanding" refers to the possibility that the intended meaning in communication may differ from the meaning that is received.

[0078] A "warning" refers to information or a message issued to draw attention to a user.

[0079] "Communication style" refers to the methods and styles of communication commonly used in a particular culture or society.

[0080] This invention provides a multi-functional interpretation system to facilitate intercultural communication. Users collect audio using earphones or a smartphone. Specifically, a wireless communication device (e.g., wireless earphones) is used as the earphone to collect audio data and transmit it to the terminal in real time.

[0081] The terminal has communication capabilities to send received audio to the server. The server converts the audio data into text using speech recognition software (e.g., a speech-to-text algorithm). The server analyzes linguistic features and intonation and uses a generation algorithm to estimate the other party's cultural background. This analysis generates a cultural profile using a general computational model.

[0082] Based on the estimated cultural background, the server translates the user's utterances into expressions appropriate to the target language's culture. This process utilizes a generative AI model, for example, using prompts such as, "Please translate this expression from French culture into a form suitable for Japanese culture."

[0083] The converted data is sent back from the server to the terminal. The terminal outputs the converted audio to the user through earphones. The user can communicate smoothly with the other party through this converted audio. Furthermore, the server monitors the context of the conversation and issues a warning to the user if it detects a topic that could be misunderstood. Specifically, an alert is generated that says, "This topic is culturally sensitive. Please choose a different topic."

[0084] This system enables users to have smooth and less misunderstanding-free conversations with people from different cultural backgrounds.

[0085] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0086] Step 1:

[0087] The user collects audio using earphones and a smartphone. The earphones capture ambient sound through a microphone and transmit it to the smartphone. Here, the input is ambient sound, and the output is digital audio data transferred to the smartphone.

[0088] Step 2:

[0089] The device compresses the collected audio data and transmits it to the server via the internet. The input is digital audio data on the smartphone, and the output is compressed audio data securely transmitted over the internet. Specifically, data compression software runs within the device.

[0090] Step 3:

[0091] The server decompresses the received compressed audio data and converts it to text using a speech recognition algorithm. The input is the compressed audio data received by the server, and the output is the text data generated after speech recognition. Advanced speech recognition software is used in this step.

[0092] Step 4:

[0093] The server uses speech analysis software to identify linguistic features and intonation from text data and sends the analysis results to a generating AI model. The input is text data generated by speech recognition, and the output is the analyzed linguistic features and intonation information. The server generates a prompt sentence, which the generating AI model then uses to operate.

[0094] Step 5:

[0095] The generative AI model estimates cultural background based on analyzed linguistic features and intonation information. The input is the analysis results sent from the server, and the output is the estimated cultural background information. Specifically, the AI ​​processes prompts such as, "Please convert this into a format suitable for Japanese culture."

[0096] Step 6:

[0097] The server converts the user's utterances into culturally appropriate expressions based on the cultural background information it receives. The input is the user's original utterance data and cultural background information, and the output is the modified utterance data. The server runs software responsible for adjusting the utterances.

[0098] Step 7:

[0099] The server converts the transformed speech data back into speech using speech synthesis technology and sends it to the terminal. The input is the user's modified text data, and the output is the data converted into a speech format.

[0100] Step 8:

[0101] The terminal sends the received audio data to the earphones and plays it back to the user. The input is the converted audio data sent from the server, and the output is the audio the user hears through the earphones. Specifically, audio playback software runs on the terminal.

[0102] Step 9:

[0103] The server monitors the conversation in real time and sends an alert to the user if it detects a risk of cultural taboos or misunderstandings. The input is continuously updated conversation data, and the output is alert information. At this stage, the warning system is active, and an audible alert is delivered through the terminal.

[0104] (Application Example 1)

[0105] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] In modern society, intercultural communication is a crucial element, but misunderstandings and inappropriate communication can occur due to differences in cultural backgrounds and values. Especially in today's world, where interaction with individuals from different cultural backgrounds is frequent within families and in daily life, there is a need for effective means to overcome these cultural barriers. Traditional interpretation systems often limit themselves to language translation, failing to adjust communication to cultural contexts.

[0107] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0108] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative model, means for adjusting cross-cultural communication and translating the user's statements, means for detecting the risk of cultural taboos and misunderstandings and issuing warnings to the user, and means for supporting dialogue between individuals with different cultural backgrounds within the household. This enables smooth intercultural communication and effectively reduces misunderstandings and communication barriers caused by cultural differences.

[0109] A "user" is an individual who uses a voice translation system to communicate across cultures.

[0110] "Voice data" refers to information obtained by collecting the voice spoken by a user using digital technology and converting it into an analyzable format.

[0111] "Analysis" is the process of identifying the linguistic characteristics and intonation of collected audio data and deriving results from that information.

[0112] A "generative model" is an algorithm that uses artificial intelligence to produce diverse outputs based on specific conditions and input information.

[0113] "Cultural background" refers to the totality of values ​​and communication styles formed from an individual's culture of origin and social environment.

[0114] "Intercultural communication" refers to the transmission of information and communication that takes place between people with different cultural backgrounds.

[0115] A "cultural taboo" refers to an action, expression, or topic that is unacceptable in a particular culture or society.

[0116] The system for implementing this invention primarily uses voice data as a medium between the user and the server. The user communicates via voice, and this voice data is collected using the microphone of a device. This device includes smartphones and home robots. The collected voice data is transmitted to the server in real time.

[0117] The server uses speech recognition software to convert speech data into text data. Technologies such as Google's Cloud Speech-to-Text API are used for this process. The server then analyzes the converted text data to identify the linguistic characteristics and intonation of the speech. Based on the analysis results, a generative AI model such as OpenAI's GPT-4 is used to estimate the cultural background of the other party.

[0118] Based on this estimation, the server transforms the user's speech to fit the identified cultural background, adjusting it to minimize the risk of cultural taboos and misunderstandings. The transformed audio is then provided to the user again as audio data and output through the user's device.

[0119] This system facilitates smooth intercultural communication within the home, avoiding taboos and misunderstandings specific to certain cultures. For example, if a foreign partner visiting a Japanese home mentions a holiday in their home country, the robot automatically translates it to a similar holiday in the host country. In this way, cultural understanding is deepened, and a better communication experience is provided.

[0120] An example of a prompt is: "Please translate the current conversation to align with the values ​​of another culture, taking cultural context into consideration. While maintaining the context of the conversation, replace it with appropriate similar holidays or topics in Japanese culture." Through this prompt, the server enables flexible communication that takes cultural context into account.

[0121] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0122] Step 1:

[0123] The user inputs voice through the device's microphone. This input is natural conversational audio data. The device collects this audio data in real time, converts it to a digital format, and sends it to a server. This conversion process includes pre-processing such as audio compression and noise reduction.

[0124] Step 2:

[0125] The server converts received audio data into text data using the Google Cloud Speech-to-Text API. Audio data is input, and text data is output. During the conversion process, phoneme analysis and natural language processing techniques are used to identify language characteristics and intonation.

[0126] Step 3:

[0127] The server uses OpenAI's GPT-4 to generate an AI model that takes the converted text data as input and estimates the other party's cultural background. This estimation extracts culture-related features from words and phrases within the text and generates an estimation result based on the prompt sentence. The output is the estimated cultural profile.

[0128] Step 4:

[0129] The server takes the estimated cultural profile into consideration and appropriately transforms the user's utterances. The input is the original text data and cultural profile, and the output is the newly text of the culturally sensitive utterance. This transformation process includes adjustments to avoid cultural taboos and misunderstandings.

[0130] Step 5:

[0131] The server generates the adjusted text data and converts it back into speech using speech synthesis technology. The generated speech data is then sent from the server to the user's terminal. A speech synthesis algorithm is used here, taking into account sound quality and fluency.

[0132] Step 6:

[0133] The user's device outputs the received audio through its speaker. This allows the user to receive culturally context-adjusted messages in audio format. The audio output also includes volume adjustment and sound quality optimization.

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] This invention combines an interpretation system that takes cultural background into account with an emotion engine. The system incorporates a device that collects audio from the user; the terminal collects the conversation between the user and the other party in real time via earphones or a smartphone. The collected audio data is transmitted to a server for analysis.

[0136] After acquiring audio data, the server performs language recognition and intonation analysis to identify the language and regional characteristics being used. Based on this information, it uses a generative model to estimate the cultural background of the conversation. Then, based on the estimated culture, the server translates the user's statements and provides interpretation using appropriate linguistic expressions. This facilitates smoother intercultural communication.

[0137] In addition, the server uses an emotion engine to recognize emotions from the user's voice. This engine analyzes the user's emotional state from factors such as tone, tempo, and volume, and evaluates stress levels and satisfaction levels. The results of this emotion analysis are reflected in the translation and warning system in real time, providing more personalized feedback.

[0138] As a concrete example, consider a conversation between a Japanese-speaking user and a Spanish-speaking person. In this case, the user's question, "Is that information interesting?", is translated to "Does it pique your interest?", taking into account the other person's cultural background and emotions, thus conveying the emotional nuance as well. Furthermore, if the emotion engine detects that the user is stressed, the server sends an alert to encourage a shift to a calmer topic to avoid excessive conversation. This kind of feedback improves both the accuracy of the translation and the user experience.

[0139] This system supports intercultural exchange by creating a more sophisticated and considerate communication environment that takes emotions and cultural backgrounds into account.

[0140] The following describes the processing flow.

[0141] Step 1:

[0142] The device collects the conversation between the user and the other party as audio data through the microphone in the earphone. The collected audio data is temporarily stored on the device.

[0143] Step 2:

[0144] The device sends the collected voice data to the server. The communication utilizes encryption protocols to protect data privacy.

[0145] Step 3:

[0146] The server processes the received audio data through a speech recognition engine to identify the language used in the conversation and further analyzes the intonation to estimate the speaker's regional background.

[0147] Step 4:

[0148] The server uses generative models based on language and intonation analysis to estimate cultural background and generate a cultural profile. This profile includes region-specific communication styles and values.

[0149] Step 5:

[0150] The server uses this cultural profile to translate the user's statements into language appropriate to the recipient's culture. The translated content is then adjusted to ensure that it is conveyed as intended to the recipient.

[0151] Step 6:

[0152] The emotion engine installed on the server analyzes the tone, tempo, and volume of the voice data to evaluate the user's emotional state in real time.

[0153] Step 7:

[0154] The server uses the emotional state evaluation obtained from the emotion engine to inform the translation, and also sends emotionally appropriate feedback and warnings to the user via voice as needed.

[0155] Step 8:

[0156] The device receives the converted translation data sent from the server and plays it back through the user's earphones.

[0157] Step 9:

[0158] Users receive feedback from the server as needed to adjust their conversations and facilitate smooth cross-cultural communication.

[0159] (Example 2)

[0160] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0161] Communication between individuals with different cultural backgrounds presents challenges, including the frequent occurrence of cultural misunderstandings and emotional discrepancies. Traditional interpretation systems are limited to pure language translation, making it difficult to adequately consider cultural backgrounds and emotions in communication. Therefore, new technologies are needed to enable smooth intercultural dialogue.

[0162] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0163] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative AI model based on the analysis results, and means for recognizing the emotional state from the voice using emotion analysis means and evaluating stress levels and satisfaction levels. This enables interpretation that takes cultural background and emotions into account, as well as individually tailored feedback.

[0164] "Means of collecting voice from users" refers to a function that uses a voice input device to acquire voices spoken by users in digital format.

[0165] "Means of analyzing audio data to identify language and intonation" refers to a function that uses speech analysis technology to identify the language used, intonation, and emphasis of speech from collected audio data.

[0166] "Methods for estimating cultural background using generative AI models" refers to a function that uses machine learning-based algorithms to infer the cultural context of a speaker from the content of audio data.

[0167] "Emotional analysis means" refers to technology that analyzes the characteristics of audio data to identify the speaker's emotional state, and to the function of scrutinizing the intensity and type of that emotion.

[0168] "Means of providing personalized feedback" refers to features that provide users with individually tailored advice and information based on analyzed cultural and emotional data.

[0169] "Means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a function that identifies misunderstandings and inappropriate cultural expressions hidden in the content and context of a conversation and informs the user of them.

[0170] This invention is a system for enabling smooth intercultural communication and includes the functions of voice collection, analysis, translation, and emotion recognition. Specific embodiments are shown below.

[0171] The user's voice is collected using the voice input device built into the device. For example, the microphone on a smartphone can be used. The collected voice data is sent to the server using a secure protocol.

[0172] The server uses the Google Cloud Speech-to-Text API or similar speech analysis services to identify language and intonation. Based on this analysis, a generative AI model is used to estimate the cultural background of the conversation. Using a general-purpose language model is effective for this purpose.

[0173] Furthermore, the server uses emotion analysis tools to analyze elements such as tone, tempo, and volume from the audio data to recognize the speaker's emotional state. For this purpose, speech analysis libraries and tools can be used.

[0174] Based on the analysis results, the server translates the user's statements into a culturally appropriate format. The translated content is sent to the terminal to support the conversation between the user and the other party.

[0175] As a concrete example, consider a scenario where a Japanese-speaking user interacts with a Spanish-speaking person. The user's statement, "Is that information interesting?", is translated into "Do you have a strong interest?" based on cultural background and sentiment analysis, and then communicated to the other person. Furthermore, if sentiment analysis determines that the user is experiencing stress, the server provides feedback encouraging them to shift to a more relaxed topic.

[0176] To effectively utilize the generative AI model, input should be in the format of an example prompt such as, "Explain how to analyze the user's voice data, taking into account cultural background and emotions, to produce an appropriate translation." Using this prompt makes it easier for the system to generate accurate translations and feedback.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The device collects the user's voice using earphones or a smartphone. The voice is converted to a digital format and encoded for transmission. The input is the user's speech, and the output is digital audio data. This is then prepared for subsequent server processing.

[0180] Step 2:

[0181] The terminal sends the collected digital audio data to the server. The data is transmitted securely using a secure protocol (e.g., HTTPS). The input is the audio data obtained in step 1, and the output is the same data received on the server side.

[0182] Step 3:

[0183] The server analyzes the received audio data to identify the language and intonation. Specifically, it uses a speech analysis API. The input is audio data, and the output is the language type and intonation pattern. Based on these results, the following processing is performed.

[0184] Step 4:

[0185] The server uses a generative AI model to estimate the cultural background based on the analysis results. The AI ​​model takes language and intonation data as input and outputs the speaker's cultural background. This process provides an appropriate context for the conversation.

[0186] Step 5:

[0187] The server processes audio data for sentiment analysis. It analyzes the tone and tempo of the voice and evaluates the emotional state. The input is audio data, and the output is an evaluation value of the emotional state (e.g., stress level or satisfaction level). This information influences the interpretation process.

[0188] Step 6:

[0189] The server translates and interprets the user's utterances, taking into account cultural background and sentiment analysis results. The appropriate expression is selected using the output of the generated AI model. The input is the user's original utterance, and the output is the translated utterance.

[0190] Step 7:

[0191] The server sends the translated utterances and feedback to the terminal. The terminal presents these to the user to aid understanding. The input is the translated utterances and feedback, and the output is what is displayed to the user.

[0192] (Application Example 2)

[0193] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0194] In intercultural communication, misunderstandings can arise not only from differences in language and cultural background, but also from subtle differences in emotional nuances. Furthermore, in multilingual environments such as movies and events, the lack of subtitles as visual information makes it difficult to accurately understand the emotions and nuances of the situation. To address these issues, a real-time interpretation system that incorporates sentiment analysis, in addition to language translation, is required.

[0195] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0196] In this invention, the server includes means for collecting speech; means for analyzing the collected speech information to identify language and intonation; means for estimating the cultural background using a generative model based on the analysis results; means for transforming speech and performing interpretation according to the estimated cultural background; means for detecting cultural taboos and risks of misunderstanding and issuing warnings to the user; means for evaluating the emotional state of the speech using sentiment analysis and reflecting the results in the interpretation; and means for generating multilingual subtitles and providing them to the user as visual information. This enables emotionally rich communication that transcends language barriers.

[0197] "Means for collecting sound" refers to a device that inputs sound data, converts it into a digital format, and supplies it to the next analysis process.

[0198] A "means for identifying language and intonation" refers to a device that has the function of analyzing and extracting the language used, as well as the intonation and tone of speech, from audio data.

[0199] "Methods for estimating cultural background using generative models" are devices that predict a speaker's cultural background by utilizing databases and algorithms based on analyzed language and intonation information.

[0200] A "means of transforming and interpreting speech" is a device that, based on an estimated cultural background, translates the original speech into appropriate linguistic expressions and mediates communication between cultures.

[0201] "A means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a device that identifies potential cultural friction points and misunderstandings that may arise during the interpretation process and informs users of them.

[0202] "A means of evaluating the emotional state of speech using emotion analysis and reflecting the results in interpretation" refers to a device that analyzes emotional information from speech and incorporates that information into the interpretation process.

[0203] "A means of generating multilingual subtitles and providing them to users as visual information" refers to a device that generates subtitles translated in real time based on audio data and presents them visually to the user.

[0204] This invention is an interpretation system designed to support smooth intercultural communication. The system is built on a cloud-based platform that runs on various devices.

[0205] System Configuration

[0206] Audio collection:

[0207] The device collects audio using microphones built into smart glasses or head-mounted displays. At this stage, the user's speech and surrounding sounds are converted into digital signals in real time and sent to a cloud server.

[0208] Voice analysis:

[0209] The server uses speech recognition engines such as Google Cloud Speech-to-Text to analyze and extract language and intonation from the input audio. This information is then used to estimate the cultural background in more detail.

[0210] Estimation of cultural background:

[0211] A generative AI model is used to estimate cultural background from the analyzed linguistic information. This estimates the speaker's cultural characteristics, and based on this, instructions are created to convert the information into appropriate linguistic expressions.

[0212] Sentiment analysis and translation:

[0213] Emotion analysis engines such as Azure® Emotion API extract emotional information from speech, and the results are then translated into the appropriate language using translation services such as the DeepL API. Emotional nuances are also reflected in the selected language during this process.

[0214] Providing visual information:

[0215] Real-time generated multilingual subtitles are displayed as visual information on a screen such as smart glasses. This display technology makes it easier for users to understand not only the language translation but also the nuances of culture and emotion.

[0216] Specific example

[0217] For example, consider a scenario at an international film festival where a film requiring multilingual support is screened. This system analyzes the film's dialogue in real time and provides translated subtitles based on cultural context and emotions. This allows audiences with different language backgrounds to experience the same emotional impact.

[0218] Example of a prompt

[0219] "Please show how to appropriately express in Japanese a scene in a film submitted to an international film festival where the protagonist is emotionally overwhelmed."

[0220] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0221] Step 1: Collect audio data

[0222] The device uses a microphone built into the smart glasses to collect user and ambient sounds. The collected analog audio is converted into a digital signal and sent to a cloud server. The input is the user's voice and ambient sounds, and the output is the digitized audio data.

[0223] Step 2: Speech Recognition and Analysis

[0224] The server uses Google Cloud Speech-to-Text to convert the transmitted audio data into text format. Next, it analyzes the language and intonation of the speech to generate foundational data for estimating the cultural background. The input is digitized audio data, and the output is the spoken content in text format, along with corresponding intonation information.

[0225] Step 3: Estimating the Cultural Background

[0226] The server uses a generative AI model to estimate cultural background from analyzed linguistic information. In this process, an estimation algorithm operates based on the generated data, extracting cultural context relevant to the country, region, or situation. The input is text and intonation data, and the output is estimated information about the cultural background.

[0227] Step 4: Conducting sentiment analysis

[0228] The server uses the Azure Emotion API to analyze and extract emotional information from audio data. This analysis generates information based on the speaker's emotional state and tone. The input is audio data, and the output is emotional information, specifically an evaluation of the speaker's stress level and satisfaction level.

[0229] Step 5: Perform translation and interpretation.

[0230] The server uses the DeepL API to translate utterances into appropriate language, taking into account cultural context and emotional information. This step specifically adjusts the translation to reflect emotions and cultural nuances. The input is cultural context and emotional information, and the output is the translated result expressed in the desired language.

[0231] Step 6: Generate and display multilingual subtitles

[0232] The device displays the translation results as visual information on smart glasses. At this stage, the information is presented as real-time subtitles, providing the user with an interactive experience that transcends language and cultural barriers. The input is translated speech data, and the output is multilingual subtitles displayed to the user.

[0233] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0234] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0235] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0236] [Second Embodiment]

[0237] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0238] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0239] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0240] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0241] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0242] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0243] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0244] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0245] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0246] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0247] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0248] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0249] This invention includes a series of devices and processes necessary to implement an interpretation system that takes cultural background into account. A device worn by the user, such as an earphone or smartphone, collects audio data. This allows the conversation between the user and the other party to be monitored in real time and transmitted to a server.

[0250] The server performs speech recognition and analysis on the received audio data to identify the language and intonation of the conversation. Using generative models based on this identified information, it estimates the other party's cultural background. This allows the server to transform the user's statements into appropriate expressions that take into account the other party's communication style and values.

[0251] The server performs a translation process according to the cultural background, sends the converted audio information back to the user's device, and outputs it through the earphones. This allows the user to receive messages in a structure appropriate to the other person's culture.

[0252] Furthermore, the server constantly monitors the context of the conversation and, if it detects a risk of cultural taboos or misunderstandings, sends an alert to the user via voice warning. This alert includes appropriate alternative expressions and suggestions for changing the topic to help the user avoid misunderstandings.

[0253] One concrete example is a conversation between a Japanese-speaking user and a French-speaking partner where a topic related to a specific expression in French culture is discussed, and it is then translated into something that is commonly discussed in Japanese culture. For instance, if a topic of a particular holiday or custom popular in France comes up, it could be changed to a discussion of a similar holiday or custom in Japan, thereby promoting cross-cultural understanding.

[0254] This system will facilitate smoother and more considerate international and intercultural communication, enabling dialogue with minimal misunderstandings.

[0255] The following describes the processing flow.

[0256] Step 1:

[0257] The device collects the conversation audio between the user and the other party in real time using the microphone in the earphone. The collected audio data is then de-noised and prepared for the next processing step as clear audio data.

[0258] Step 2:

[0259] The terminal sends the pre-processed audio data to the server via a network such as the internet. The network connection utilizes encryption protocols for data protection.

[0260] Step 3:

[0261] The server places the received audio data into a queue for analysis. During this process, speech recognition technology is used to identify the language being used in the audio data.

[0262] Step 4:

[0263] The server analyzes the intonation of the audio data to estimate the speaker's regional or cultural background. It identifies geographical features based on tone, rhythm, and accent.

[0264] Step 5:

[0265] The server uses a generative model to generate cultural profiles based on identified cultural elements, such as politeness and directness.

[0266] Step 6:

[0267] Based on the generated cultural profile, the server translates the user's utterances into language that is easily understood by the recipient. This translation is adjusted to match the selected context.

[0268] Step 7:

[0269] The server sends the converted language data back to the terminal. The terminal converts this data into an audio signal and transmits it to the user in real time via earphones.

[0270] Step 8:

[0271] The server performs continuous response analysis to detect potential cultural taboos and misunderstandings during conversations. If a risk is detected, the server sends an audio warning to the user.

[0272] Step 9:

[0273] The server generates alternative expressions and suggestions for new topics based on the risk of misunderstanding, and provides these to the user along with alerts. This helps the user to appropriately adjust the conversation.

[0274] (Example 1)

[0275] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0276] In today's global society, opportunities for people with different languages ​​and cultural backgrounds to communicate are increasing. In these situations, not only language barriers but also cultural misunderstandings and taboos can become major obstacles. Traditional interpretation systems focus on language conversion, but they often lack sufficient consideration and adaptation to cultural backgrounds, potentially hindering smooth communication and leading to misunderstandings.

[0277] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0278] In this invention, the server includes means for collecting auditory information from the user, means for analyzing the collected auditory information to identify linguistic features and intonation, and means for estimating the cultural background using a generation algorithm based on the analysis results. This enables smooth intercultural communication by providing appropriate interpretation and warnings while considering the cultural context.

[0279] "User" refers to a person who uses the system to collect and interpret audio data.

[0280] "Auditory information" refers to data that includes audio data and language information collected from users.

[0281] "Linguistic features" refer to the types of language, unique expressions, and vocabulary patterns extracted from audio data.

[0282] "Pitch" refers to elements indicating the characteristics of speech sounds in language, such as intonation, stress, and rhythm of speech.

[0283] "Generation algorithm" means a computational procedure or model used to generate or transform information.

[0284] "Cultural background" refers to values, beliefs, norms, etc. that are generally shared in a specific culture or society.

[0285] "Interpretation" refers to an act of promoting mutual understanding by appropriately conveying the content expressed in one language into another language.

[0286] "Cultural taboo" means actions, expressions, topics, etc. that should be avoided in a specific culture.

[0287] "Risk of misunderstanding" refers to the possibility that the intended meaning and the received meaning in communication are different.

[0288] "Warning" means information or messages issued to prompt users to pay attention.

[0289] "Communication style" refers to the methods or styles of communication generally used in a specific culture or society.

[0290] The present invention provides a multifunctional interpretation system for facilitating communication between different cultures. Users collect voices using earphones or smartphones. Specifically, a device capable of wireless communication (for example, wireless earphones) is used as the earphones to collect voice data and transmit it to the terminal in real time.

[0291] The terminal has communication capabilities to send received audio to the server. The server converts the audio data into text using speech recognition software (e.g., a speech-to-text algorithm). The server analyzes linguistic features and intonation and uses a generation algorithm to estimate the other party's cultural background. This analysis generates a cultural profile using a general computational model.

[0292] Based on the estimated cultural background, the server translates the user's utterances into expressions appropriate to the target language's culture. This process utilizes a generative AI model, for example, using prompts such as, "Please translate this expression from French culture into a form suitable for Japanese culture."

[0293] The converted data is sent back from the server to the terminal. The terminal outputs the converted audio to the user through earphones. The user can communicate smoothly with the other party through this converted audio. Furthermore, the server monitors the context of the conversation and issues a warning to the user if it detects a topic that could be misunderstood. Specifically, an alert is generated that says, "This topic is culturally sensitive. Please choose a different topic."

[0294] This system enables users to have smooth and less misunderstanding-free conversations with people from different cultural backgrounds.

[0295] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0296] Step 1:

[0297] The user collects audio using earphones and a smartphone. The earphones capture ambient sound through a microphone and transmit it to the smartphone. Here, the input is ambient sound, and the output is digital audio data transferred to the smartphone.

[0298] Step 2:

[0299] The terminal compresses the collected voice data and transmits it to the server via the Internet. The input is digital voice data on a smartphone, and the output is compressed voice data that is securely transmitted through the Internet. As a specific operation, data compression software operates inside the terminal.

[0300] Step 3:

[0301] The server decompresses the received compressed voice data and converts it into text using a voice recognition algorithm. The input is the compressed voice data that has reached the server, and the output is the text data generated through voice recognition. In this step, advanced voice recognition software is used.

[0302] Step 4:

[0303] The server uses voice analysis software to identify language features and tones from the text data and transmits the analysis results to the generative AI model. The input is the text data generated by voice recognition, and the output is the analyzed language features and tone information. The server generates a prompt sentence, and based on that, the generative AI model operates.

[0304] Step 5:

[0305] The generative AI model estimates the cultural background based on the analyzed language features and tone information. The input is the analysis results transmitted from the server, and the output is the estimated cultural background information. As a specific operation, the AI processes a prompt sentence such as "Please convert it into a form suitable for Japanese culture."

[0306] Step 6:

[0307] Based on the received cultural background information, the server converts the user's speech into a culturally appropriate expression. The input is the original speech data of the user and the cultural background information, and the output is the corrected speech data. The server executes software responsible for adjusting the speech.

[0308] Step 7:

[0309] The server converts the transformed speech data back into speech using speech synthesis technology and sends it to the terminal. The input is the user's modified text data, and the output is the data converted into a speech format.

[0310] Step 8:

[0311] The terminal sends the received audio data to the earphones and plays it back to the user. The input is the converted audio data sent from the server, and the output is the audio the user hears through the earphones. Specifically, audio playback software runs on the terminal.

[0312] Step 9:

[0313] The server monitors the conversation in real time and sends an alert to the user if it detects a risk of cultural taboos or misunderstandings. The input is continuously updated conversation data, and the output is alert information. At this stage, the warning system is active, and an audible alert is delivered through the terminal.

[0314] (Application Example 1)

[0315] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0316] In modern society, intercultural communication is a crucial element, but misunderstandings and inappropriate communication can occur due to differences in cultural backgrounds and values. Especially in today's world, where interaction with individuals from different cultural backgrounds is frequent within families and in daily life, there is a need for effective means to overcome these cultural barriers. Traditional interpretation systems often limit themselves to language translation, failing to adjust communication to cultural contexts.

[0317] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0318] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative model, means for adjusting cross-cultural communication and translating the user's statements, means for detecting the risk of cultural taboos and misunderstandings and issuing warnings to the user, and means for supporting dialogue between individuals with different cultural backgrounds within the household. This enables smooth intercultural communication and effectively reduces misunderstandings and communication barriers caused by cultural differences.

[0319] A "user" is an individual who uses a voice translation system to communicate across cultures.

[0320] "Voice data" refers to information obtained by collecting the voice spoken by a user using digital technology and converting it into an analyzable format.

[0321] "Analysis" is the process of identifying the linguistic characteristics and intonation of collected audio data and deriving results from that information.

[0322] A "generative model" is an algorithm that uses artificial intelligence to produce diverse outputs based on specific conditions and input information.

[0323] "Cultural background" refers to the totality of values ​​and communication styles formed from an individual's culture of origin and social environment.

[0324] "Intercultural communication" refers to the transmission of information and communication that takes place between people with different cultural backgrounds.

[0325] A "cultural taboo" refers to an action, expression, or topic that is unacceptable in a particular culture or society.

[0326] The system for implementing this invention primarily uses voice data as a medium between the user and the server. The user communicates via voice, and this voice data is collected using the microphone of a device. This device includes smartphones and home robots. The collected voice data is transmitted to the server in real time.

[0327] The server uses speech recognition software to convert audio data into text data. Technologies such as the Google Cloud Speech-to-Text API are used for this process. The server then analyzes the converted text data to identify the linguistic characteristics and intonation of the speech. Based on the analysis results, a generative AI model such as OpenAI's GPT-4 is used to estimate the cultural background of the other party.

[0328] Based on this estimation, the server transforms the user's speech to fit the identified cultural background, adjusting it to minimize the risk of cultural taboos and misunderstandings. The transformed audio is then provided to the user again as audio data and output through the user's device.

[0329] This system facilitates smooth intercultural communication within the home, avoiding taboos and misunderstandings specific to certain cultures. For example, if a foreign partner visiting a Japanese home mentions a holiday in their home country, the robot automatically translates it to a similar holiday in the host country. In this way, cultural understanding is deepened, and a better communication experience is provided.

[0330] An example of a prompt is: "Please translate the current conversation to align with the values ​​of another culture, taking cultural context into consideration. While maintaining the context of the conversation, replace it with appropriate similar holidays or topics in Japanese culture." Through this prompt, the server enables flexible communication that takes cultural context into account.

[0331] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0332] Step 1:

[0333] The user inputs voice through the device's microphone. This input is natural conversational audio data. The device collects this audio data in real time, converts it to a digital format, and sends it to a server. This conversion process includes pre-processing such as audio compression and noise reduction.

[0334] Step 2:

[0335] The server converts received audio data into text data using the Google Cloud Speech-to-Text API. Audio data is input, and text data is output. During the conversion process, phoneme analysis and natural language processing techniques are used to identify language characteristics and intonation.

[0336] Step 3:

[0337] The server uses OpenAI's GPT-4 to generate an AI model that takes the converted text data as input and estimates the other party's cultural background. This estimation extracts culture-related features from words and phrases within the text and generates an estimation result based on the prompt sentence. The output is the estimated cultural profile.

[0338] Step 4:

[0339] The server takes the estimated cultural profile into consideration and appropriately transforms the user's utterances. The input is the original text data and cultural profile, and the output is the newly text of the culturally sensitive utterance. This transformation process includes adjustments to avoid cultural taboos and misunderstandings.

[0340] Step 5:

[0341] The server generates the adjusted text data and converts it back into speech using speech synthesis technology. The generated speech data is then sent from the server to the user's terminal. A speech synthesis algorithm is used here, taking into account sound quality and fluency.

[0342] Step 6:

[0343] The user's device outputs the received audio through its speaker. This allows the user to receive culturally context-adjusted messages in audio format. The audio output also includes volume adjustment and sound quality optimization.

[0344] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0345] This invention combines an interpretation system that takes cultural background into account with an emotion engine. The system incorporates a device that collects audio from the user; the terminal collects the conversation between the user and the other party in real time via earphones or a smartphone. The collected audio data is transmitted to a server for analysis.

[0346] After acquiring audio data, the server performs language recognition and intonation analysis to identify the language and regional characteristics being used. Based on this information, it uses a generative model to estimate the cultural background of the conversation. Then, based on the estimated culture, the server translates the user's statements and provides interpretation using appropriate linguistic expressions. This facilitates smoother intercultural communication.

[0347] In addition, the server uses an emotion engine to recognize emotions from the user's voice. This engine analyzes the user's emotional state from factors such as tone, tempo, and volume, and evaluates stress levels and satisfaction levels. The results of this emotion analysis are reflected in the translation and warning system in real time, providing more personalized feedback.

[0348] As a concrete example, consider a conversation between a Japanese-speaking user and a Spanish-speaking person. In this case, the user's question, "Is that information interesting?", is translated to "Does it pique your interest?", taking into account the other person's cultural background and emotions, thus conveying the emotional nuance as well. Furthermore, if the emotion engine detects that the user is stressed, the server sends an alert to encourage a shift to a calmer topic to avoid excessive conversation. This kind of feedback improves both the accuracy of the translation and the user experience.

[0349] This system supports intercultural exchange by creating a more sophisticated and considerate communication environment that takes emotions and cultural backgrounds into account.

[0350] The following describes the processing flow.

[0351] Step 1:

[0352] The device collects the conversation between the user and the other party as audio data through the microphone in the earphone. The collected audio data is temporarily stored on the device.

[0353] Step 2:

[0354] The device sends the collected voice data to the server. The communication utilizes encryption protocols to protect data privacy.

[0355] Step 3:

[0356] The server processes the received audio data through a speech recognition engine to identify the language used in the conversation and further analyzes the intonation to estimate the speaker's regional background.

[0357] Step 4:

[0358] The server uses generative models based on language and intonation analysis to estimate cultural background and generate a cultural profile. This profile includes region-specific communication styles and values.

[0359] Step 5:

[0360] The server uses this cultural profile to translate the user's statements into language appropriate to the recipient's culture. The translated content is then adjusted to ensure that it is conveyed as intended to the recipient.

[0361] Step 6:

[0362] The emotion engine installed on the server analyzes the tone, tempo, and volume of the voice data to evaluate the user's emotional state in real time.

[0363] Step 7:

[0364] The server uses the emotional state evaluation obtained from the emotion engine to inform the translation, and also sends emotionally appropriate feedback and warnings to the user via voice as needed.

[0365] Step 8:

[0366] The device receives the converted translation data sent from the server and plays it back through the user's earphones.

[0367] Step 9:

[0368] Users receive feedback from the server as needed to adjust their conversations and facilitate smooth cross-cultural communication.

[0369] (Example 2)

[0370] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0371] Communication between individuals with different cultural backgrounds presents challenges, including the frequent occurrence of cultural misunderstandings and emotional discrepancies. Traditional interpretation systems are limited to pure language translation, making it difficult to adequately consider cultural backgrounds and emotions in communication. Therefore, new technologies are needed to enable smooth intercultural dialogue.

[0372] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0373] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative AI model based on the analysis results, and means for recognizing the emotional state from the voice using emotion analysis means and evaluating stress levels and satisfaction levels. This enables interpretation that takes cultural background and emotions into account, as well as individually tailored feedback.

[0374] "Means of collecting voice from users" refers to a function that uses a voice input device to acquire voices spoken by users in digital format.

[0375] "Means of analyzing audio data to identify language and intonation" refers to a function that uses speech analysis technology to identify the language used, intonation, and emphasis of speech from collected audio data.

[0376] "Methods for estimating cultural background using generative AI models" refers to a function that uses machine learning-based algorithms to infer the cultural context of a speaker from the content of audio data.

[0377] "Emotional analysis means" refers to technology that analyzes the characteristics of audio data to identify the speaker's emotional state, and to the function of scrutinizing the intensity and type of that emotion.

[0378] "Means of providing personalized feedback" refers to features that provide users with individually tailored advice and information based on analyzed cultural and emotional data.

[0379] "Means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a function that identifies misunderstandings and inappropriate cultural expressions hidden in the content and context of a conversation and informs the user of them.

[0380] This invention is a system for enabling smooth intercultural communication and includes the functions of voice collection, analysis, translation, and emotion recognition. Specific embodiments are shown below.

[0381] The user's voice is collected using the voice input device built into the device. For example, the microphone on a smartphone can be used. The collected voice data is sent to the server using a secure protocol.

[0382] The server uses the Google Cloud Speech-to-Text API or similar speech analysis services to identify language and intonation. Based on this analysis, a generative AI model is used to estimate the cultural background of the conversation. Using a general-purpose language model is effective for this purpose.

[0383] Furthermore, the server uses emotion analysis tools to analyze elements such as tone, tempo, and volume from the audio data to recognize the speaker's emotional state. For this purpose, speech analysis libraries and tools can be used.

[0384] Based on the analysis results, the server translates the user's statements into a culturally appropriate format. The translated content is sent to the terminal to support the conversation between the user and the other party.

[0385] As a concrete example, consider a scenario where a Japanese-speaking user interacts with a Spanish-speaking person. The user's statement, "Is that information interesting?", is translated into "Do you have a strong interest?" based on cultural background and sentiment analysis, and then communicated to the other person. Furthermore, if sentiment analysis determines that the user is experiencing stress, the server provides feedback encouraging them to shift to a more relaxed topic.

[0386] To effectively utilize the generative AI model, input should be in the format of an example prompt such as, "Explain how to analyze the user's voice data, taking into account cultural background and emotions, to produce an appropriate translation." Using this prompt makes it easier for the system to generate accurate translations and feedback.

[0387] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0388] Step 1:

[0389] The device collects the user's voice using earphones or a smartphone. The voice is converted to a digital format and encoded for transmission. The input is the user's speech, and the output is digital audio data. This is then prepared for subsequent server processing.

[0390] Step 2:

[0391] The terminal sends the collected digital audio data to the server. The data is transmitted securely using a secure protocol (e.g., HTTPS). The input is the audio data obtained in step 1, and the output is the same data received on the server side.

[0392] Step 3:

[0393] The server analyzes the received audio data to identify the language and intonation. Specifically, it uses a speech analysis API. The input is audio data, and the output is the language type and intonation pattern. Based on these results, the following processing is performed.

[0394] Step 4:

[0395] The server uses a generative AI model to estimate the cultural background based on the analysis results. The AI ​​model takes language and intonation data as input and outputs the speaker's cultural background. This process provides an appropriate context for the conversation.

[0396] Step 5:

[0397] The server processes audio data for sentiment analysis. It analyzes the tone and tempo of the voice and evaluates the emotional state. The input is audio data, and the output is an evaluation value of the emotional state (e.g., stress level or satisfaction level). This information influences the interpretation process.

[0398] Step 6:

[0399] The server translates and interprets the user's utterances, taking into account cultural background and sentiment analysis results. The appropriate expression is selected using the output of the generated AI model. The input is the user's original utterance, and the output is the translated utterance.

[0400] Step 7:

[0401] The server sends the translated utterances and feedback to the terminal. The terminal presents these to the user to aid understanding. The input is the translated utterances and feedback, and the output is what is displayed to the user.

[0402] (Application Example 2)

[0403] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0404] In intercultural communication, misunderstandings can arise not only from differences in language and cultural background, but also from subtle differences in emotional nuances. Furthermore, in multilingual environments such as movies and events, the lack of subtitles as visual information makes it difficult to accurately understand the emotions and nuances of the situation. To address these issues, a real-time interpretation system that incorporates sentiment analysis, in addition to language translation, is required.

[0405] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0406] In this invention, the server includes means for collecting speech; means for analyzing the collected speech information to identify language and intonation; means for estimating the cultural background using a generative model based on the analysis results; means for transforming speech and performing interpretation according to the estimated cultural background; means for detecting cultural taboos and risks of misunderstanding and issuing warnings to the user; means for evaluating the emotional state of the speech using sentiment analysis and reflecting the results in the interpretation; and means for generating multilingual subtitles and providing them to the user as visual information. This enables emotionally rich communication that transcends language barriers.

[0407] "Means for collecting sound" refers to a device that inputs sound data, converts it into a digital format, and supplies it to the next analysis process.

[0408] A "means for identifying language and intonation" refers to a device that has the function of analyzing and extracting the language used, as well as the intonation and tone of speech, from audio data.

[0409] "Methods for estimating cultural background using generative models" are devices that predict a speaker's cultural background by utilizing databases and algorithms based on analyzed language and intonation information.

[0410] A "means of transforming and interpreting speech" is a device that, based on an estimated cultural background, translates the original speech into appropriate linguistic expressions and mediates communication between cultures.

[0411] "A means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a device that identifies potential cultural friction points and misunderstandings that may arise during the interpretation process and informs users of them.

[0412] "A means of evaluating the emotional state of speech using emotion analysis and reflecting the results in interpretation" refers to a device that analyzes emotional information from speech and incorporates that information into the interpretation process.

[0413] "A means of generating multilingual subtitles and providing them to users as visual information" refers to a device that generates subtitles translated in real time based on audio data and presents them visually to the user.

[0414] This invention is an interpretation system designed to support smooth intercultural communication. The system is built on a cloud-based platform that runs on various devices.

[0415] System Configuration

[0416] Audio collection:

[0417] The device collects audio using microphones built into smart glasses or head-mounted displays. At this stage, the user's speech and surrounding sounds are converted into digital signals in real time and sent to a cloud server.

[0418] Voice analysis:

[0419] The server uses speech recognition engines such as Google Cloud Speech-to-Text to analyze and extract language and intonation from the input audio. This information is then used to estimate the cultural background in more detail.

[0420] Estimation of cultural background:

[0421] A generative AI model is used to estimate cultural background from the analyzed linguistic information. This estimates the speaker's cultural characteristics, and based on this, instructions are created to convert the information into appropriate linguistic expressions.

[0422] Sentiment analysis and translation:

[0423] Emotion analysis engines such as the Azure Emotion API extract emotional information from speech, and the results are then translated into the appropriate language using translation services such as the DeepL API. During this process, emotional nuances are also reflected in the selected language.

[0424] Providing visual information:

[0425] Real-time generated multilingual subtitles are displayed as visual information on a screen such as smart glasses. This display technology makes it easier for users to understand not only the language translation but also the nuances of culture and emotion.

[0426] Specific example

[0427] For example, consider a scenario at an international film festival where a film requiring multilingual support is screened. This system analyzes the film's dialogue in real time and provides translated subtitles based on cultural context and emotions. This allows audiences with different language backgrounds to experience the same emotional impact.

[0428] Example of a prompt

[0429] "Please show how to appropriately express in Japanese a scene in a film submitted to an international film festival where the protagonist is emotionally overwhelmed."

[0430] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0431] Step 1: Collect audio data

[0432] The device uses a microphone built into the smart glasses to collect user and ambient sounds. The collected analog audio is converted into a digital signal and sent to a cloud server. The input is the user's voice and ambient sounds, and the output is the digitized audio data.

[0433] Step 2: Speech Recognition and Analysis

[0434] The server uses Google Cloud Speech-to-Text to convert the transmitted audio data into text format. Next, it analyzes the language and intonation of the speech to generate foundational data for estimating the cultural background. The input is digitized audio data, and the output is the spoken content in text format, along with corresponding intonation information.

[0435] Step 3: Estimating the Cultural Background

[0436] The server uses a generative AI model to estimate cultural background from analyzed linguistic information. In this process, an estimation algorithm operates based on the generated data, extracting cultural context relevant to the country, region, or situation. The input is text and intonation data, and the output is estimated information about the cultural background.

[0437] Step 4: Conducting sentiment analysis

[0438] The server uses the Azure Emotion API to analyze and extract emotional information from audio data. This analysis generates information based on the speaker's emotional state and tone. The input is audio data, and the output is emotional information, specifically an evaluation of the speaker's stress level and satisfaction level.

[0439] Step 5: Perform translation and interpretation.

[0440] The server uses the DeepL API to translate utterances into appropriate language, taking into account cultural context and emotional information. This step specifically adjusts the translation to reflect emotions and cultural nuances. The input is cultural context and emotional information, and the output is the translated result expressed in the desired language.

[0441] Step 6: Generate and display multilingual subtitles

[0442] The device displays the translation results as visual information on smart glasses. At this stage, the information is presented as real-time subtitles, providing the user with an interactive experience that transcends language and cultural barriers. The input is translated speech data, and the output is multilingual subtitles displayed to the user.

[0443] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0444] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0445] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0446] [Third Embodiment]

[0447] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0448] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0449] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0450] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0451] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0452] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0453] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0454] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0455] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0456] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0457] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0458] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0459] This invention includes a series of devices and processes necessary to implement an interpretation system that takes cultural background into account. A device worn by the user, such as an earphone or smartphone, collects audio data. This allows the conversation between the user and the other party to be monitored in real time and transmitted to a server.

[0460] The server performs speech recognition and analysis on the received audio data to identify the language and intonation of the conversation. Using generative models based on this identified information, it estimates the other party's cultural background. This allows the server to transform the user's statements into appropriate expressions that take into account the other party's communication style and values.

[0461] The server performs a translation process according to the cultural background, sends the converted audio information back to the user's device, and outputs it through the earphones. This allows the user to receive messages in a structure appropriate to the other person's culture.

[0462] Furthermore, the server constantly monitors the context of the conversation and, if it detects a risk of cultural taboos or misunderstandings, sends an alert to the user via voice warning. This alert includes appropriate alternative expressions and suggestions for changing the topic to help the user avoid misunderstandings.

[0463] One concrete example is a conversation between a Japanese-speaking user and a French-speaking partner where a topic related to a specific expression in French culture is discussed, and it is then translated into something that is commonly discussed in Japanese culture. For instance, if a topic of a particular holiday or custom popular in France comes up, it could be changed to a discussion of a similar holiday or custom in Japan, thereby promoting cross-cultural understanding.

[0464] This system will facilitate smoother and more considerate international and intercultural communication, enabling dialogue with minimal misunderstandings.

[0465] The following describes the processing flow.

[0466] Step 1:

[0467] The device collects the conversation audio between the user and the other party in real time using the microphone in the earphone. The collected audio data is then de-noised and prepared for the next processing step as clear audio data.

[0468] Step 2:

[0469] The terminal sends the pre-processed audio data to the server via a network such as the internet. The network connection utilizes encryption protocols for data protection.

[0470] Step 3:

[0471] The server places the received audio data into a queue for analysis. During this process, speech recognition technology is used to identify the language being used in the audio data.

[0472] Step 4:

[0473] The server analyzes the intonation of the audio data to estimate the speaker's regional or cultural background. It identifies geographical features based on tone, rhythm, and accent.

[0474] Step 5:

[0475] The server uses a generative model to generate cultural profiles based on identified cultural elements, such as politeness and directness.

[0476] Step 6:

[0477] Based on the generated cultural profile, the server translates the user's utterances into language that is easily understood by the recipient. This translation is adjusted to match the selected context.

[0478] Step 7:

[0479] The server sends the converted language data back to the terminal. The terminal converts this data into an audio signal and transmits it to the user in real time via earphones.

[0480] Step 8:

[0481] The server performs continuous response analysis to detect potential cultural taboos and misunderstandings during conversations. If a risk is detected, the server sends an audio warning to the user.

[0482] Step 9:

[0483] The server generates alternative expressions and suggestions for new topics based on the risk of misunderstanding, and provides these to the user along with alerts. This helps the user to appropriately adjust the conversation.

[0484] (Example 1)

[0485] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0486] In today's global society, opportunities for people with different languages ​​and cultural backgrounds to communicate are increasing. In these situations, not only language barriers but also cultural misunderstandings and taboos can become major obstacles. Traditional interpretation systems focus on language conversion, but they often lack sufficient consideration and adaptation to cultural backgrounds, potentially hindering smooth communication and leading to misunderstandings.

[0487] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0488] In this invention, the server includes means for collecting auditory information from the user, means for analyzing the collected auditory information to identify linguistic features and intonation, and means for estimating the cultural background using a generation algorithm based on the analysis results. This enables smooth intercultural communication by providing appropriate interpretation and warnings while considering the cultural context.

[0489] "User" refers to a person who uses the system to collect and interpret audio data.

[0490] "Auditory information" refers to data that includes audio data and language information collected from users.

[0491] "Linguistic features" refer to the types of language, unique expressions, and vocabulary patterns extracted from audio data.

[0492] "Intonation" refers to elements that describe the characteristics of speech in language, such as intonation, volume, and rhythm.

[0493] A "generative algorithm" refers to a set of computational procedures or models used to generate or transform information.

[0494] "Cultural background" refers to the values, beliefs, and norms that are generally shared within a particular culture or society.

[0495] "Interpretation" refers to the act of facilitating mutual understanding by appropriately conveying content expressed in one language into another language.

[0496] A "cultural taboo" refers to actions, expressions, or topics that are considered to be avoided in a particular culture.

[0497] The "risk of misunderstanding" refers to the possibility that the intended meaning in communication may differ from the meaning that is received.

[0498] A "warning" refers to information or a message issued to draw attention to a user.

[0499] "Communication style" refers to the methods and styles of communication commonly used in a particular culture or society.

[0500] This invention provides a multi-functional interpretation system to facilitate intercultural communication. Users collect audio using earphones or a smartphone. Specifically, a wireless communication device (e.g., wireless earphones) is used as the earphone to collect audio data and transmit it to the terminal in real time.

[0501] The terminal has communication capabilities to send received audio to the server. The server converts the audio data into text using speech recognition software (e.g., a speech-to-text algorithm). The server analyzes linguistic features and intonation and uses a generation algorithm to estimate the other party's cultural background. This analysis generates a cultural profile using a general computational model.

[0502] Based on the estimated cultural background, the server translates the user's utterances into expressions appropriate to the target language's culture. This process utilizes a generative AI model, for example, using prompts such as, "Please translate this expression from French culture into a form suitable for Japanese culture."

[0503] The converted data is sent back from the server to the terminal. The terminal outputs the converted audio to the user through earphones. The user can communicate smoothly with the other party through this converted audio. Furthermore, the server monitors the context of the conversation and issues a warning to the user if it detects a topic that could be misunderstood. Specifically, an alert is generated that says, "This topic is culturally sensitive. Please choose a different topic."

[0504] This system enables users to have smooth and less misunderstanding-free conversations with people from different cultural backgrounds.

[0505] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0506] Step 1:

[0507] The user collects audio using earphones and a smartphone. The earphones capture ambient sound through a microphone and transmit it to the smartphone. Here, the input is ambient sound, and the output is digital audio data transferred to the smartphone.

[0508] Step 2:

[0509] The device compresses the collected audio data and transmits it to the server via the internet. The input is digital audio data on the smartphone, and the output is compressed audio data securely transmitted over the internet. Specifically, data compression software runs within the device.

[0510] Step 3:

[0511] The server decompresses the received compressed audio data and converts it to text using a speech recognition algorithm. The input is the compressed audio data received by the server, and the output is the text data generated after speech recognition. Advanced speech recognition software is used in this step.

[0512] Step 4:

[0513] The server uses speech analysis software to identify linguistic features and intonation from text data and sends the analysis results to a generating AI model. The input is text data generated by speech recognition, and the output is the analyzed linguistic features and intonation information. The server generates a prompt sentence, which the generating AI model then uses to operate.

[0514] Step 5:

[0515] The generative AI model estimates cultural background based on analyzed linguistic features and intonation information. The input is the analysis results sent from the server, and the output is the estimated cultural background information. Specifically, the AI ​​processes prompts such as, "Please convert this into a format suitable for Japanese culture."

[0516] Step 6:

[0517] The server converts the user's utterances into culturally appropriate expressions based on the cultural background information it receives. The input is the user's original utterance data and cultural background information, and the output is the modified utterance data. The server runs software responsible for adjusting the utterances.

[0518] Step 7:

[0519] The server converts the transformed speech data back into speech using speech synthesis technology and sends it to the terminal. The input is the user's modified text data, and the output is the data converted into a speech format.

[0520] Step 8:

[0521] The terminal sends the received audio data to the earphones and plays it back to the user. The input is the converted audio data sent from the server, and the output is the audio the user hears through the earphones. Specifically, audio playback software runs on the terminal.

[0522] Step 9:

[0523] The server monitors the conversation in real time and sends an alert to the user if it detects a risk of cultural taboos or misunderstandings. The input is continuously updated conversation data, and the output is alert information. At this stage, the warning system is active, and an audible alert is delivered through the terminal.

[0524] (Application Example 1)

[0525] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0526] In modern society, intercultural communication is a crucial element, but misunderstandings and inappropriate communication can occur due to differences in cultural backgrounds and values. Especially in today's world, where interaction with individuals from different cultural backgrounds is frequent within families and in daily life, there is a need for effective means to overcome these cultural barriers. Traditional interpretation systems often limit themselves to language translation, failing to adjust communication to cultural contexts.

[0527] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0528] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative model, means for adjusting cross-cultural communication and translating the user's statements, means for detecting the risk of cultural taboos and misunderstandings and issuing warnings to the user, and means for supporting dialogue between individuals with different cultural backgrounds within the household. This enables smooth intercultural communication and effectively reduces misunderstandings and communication barriers caused by cultural differences.

[0529] A "user" is an individual who uses a voice translation system to communicate across cultures.

[0530] "Voice data" refers to information obtained by collecting the voice spoken by a user using digital technology and converting it into an analyzable format.

[0531] "Analysis" is the process of identifying the linguistic characteristics and intonation of collected audio data and deriving results from that information.

[0532] A "generative model" is an algorithm that uses artificial intelligence to produce diverse outputs based on specific conditions and input information.

[0533] "Cultural background" refers to the totality of values ​​and communication styles formed from an individual's culture of origin and social environment.

[0534] "Intercultural communication" refers to the transmission of information and communication that takes place between people with different cultural backgrounds.

[0535] A "cultural taboo" refers to an action, expression, or topic that is unacceptable in a particular culture or society.

[0536] The system for implementing this invention primarily uses voice data as a medium between the user and the server. The user communicates via voice, and this voice data is collected using the microphone of a device. This device includes smartphones and home robots. The collected voice data is transmitted to the server in real time.

[0537] The server uses speech recognition software to convert audio data into text data. Technologies such as the Google Cloud Speech-to-Text API are used for this process. The server then analyzes the converted text data to identify the linguistic characteristics and intonation of the speech. Based on the analysis results, a generative AI model such as OpenAI's GPT-4 is used to estimate the cultural background of the other party.

[0538] Based on this estimation, the server transforms the user's speech to fit the identified cultural background, adjusting it to minimize the risk of cultural taboos and misunderstandings. The transformed audio is then provided to the user again as audio data and output through the user's device.

[0539] This system facilitates smooth intercultural communication within the home, avoiding taboos and misunderstandings specific to certain cultures. For example, if a foreign partner visiting a Japanese home mentions a holiday in their home country, the robot automatically translates it to a similar holiday in the host country. In this way, cultural understanding is deepened, and a better communication experience is provided.

[0540] An example of a prompt is: "Please translate the current conversation to align with the values ​​of another culture, taking cultural context into consideration. While maintaining the context of the conversation, replace it with appropriate similar holidays or topics in Japanese culture." Through this prompt, the server enables flexible communication that takes cultural context into account.

[0541] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0542] Step 1:

[0543] The user inputs voice through the device's microphone. This input is natural conversational audio data. The device collects this audio data in real time, converts it to a digital format, and sends it to a server. This conversion process includes pre-processing such as audio compression and noise reduction.

[0544] Step 2:

[0545] The server converts received audio data into text data using the Google Cloud Speech-to-Text API. Audio data is input, and text data is output. During the conversion process, phoneme analysis and natural language processing techniques are used to identify language characteristics and intonation.

[0546] Step 3:

[0547] The server uses OpenAI's GPT-4 to generate an AI model that takes the converted text data as input and estimates the other party's cultural background. This estimation extracts culture-related features from words and phrases within the text and generates an estimation result based on the prompt sentence. The output is the estimated cultural profile.

[0548] Step 4:

[0549] The server takes the estimated cultural profile into consideration and appropriately transforms the user's utterances. The input is the original text data and cultural profile, and the output is the newly text of the culturally sensitive utterance. This transformation process includes adjustments to avoid cultural taboos and misunderstandings.

[0550] Step 5:

[0551] The server generates the adjusted text data and converts it back into speech using speech synthesis technology. The generated speech data is then sent from the server to the user's terminal. A speech synthesis algorithm is used here, taking into account sound quality and fluency.

[0552] Step 6:

[0553] The user's device outputs the received audio through its speaker. This allows the user to receive culturally context-adjusted messages in audio format. The audio output also includes volume adjustment and sound quality optimization.

[0554] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0555] This invention combines an interpretation system that takes cultural background into account with an emotion engine. The system incorporates a device that collects audio from the user; the terminal collects the conversation between the user and the other party in real time via earphones or a smartphone. The collected audio data is transmitted to a server for analysis.

[0556] After acquiring audio data, the server performs language recognition and intonation analysis to identify the language and regional characteristics being used. Based on this information, it uses a generative model to estimate the cultural background of the conversation. Then, based on the estimated culture, the server translates the user's statements and provides interpretation using appropriate linguistic expressions. This facilitates smoother intercultural communication.

[0557] In addition, the server uses an emotion engine to recognize emotions from the user's voice. This engine analyzes the user's emotional state from factors such as tone, tempo, and volume, and evaluates stress levels and satisfaction levels. The results of this emotion analysis are reflected in the translation and warning system in real time, providing more personalized feedback.

[0558] As a concrete example, consider a conversation between a Japanese-speaking user and a Spanish-speaking person. In this case, the user's question, "Is that information interesting?", is translated to "Does it pique your interest?", taking into account the other person's cultural background and emotions, thus conveying the emotional nuance as well. Furthermore, if the emotion engine detects that the user is stressed, the server sends an alert to encourage a shift to a calmer topic to avoid excessive conversation. This kind of feedback improves both the accuracy of the translation and the user experience.

[0559] This system supports intercultural exchange by creating a more sophisticated and considerate communication environment that takes emotions and cultural backgrounds into account.

[0560] The following describes the processing flow.

[0561] Step 1:

[0562] The device collects the conversation between the user and the other party as audio data through the microphone in the earphone. The collected audio data is temporarily stored on the device.

[0563] Step 2:

[0564] The device sends the collected voice data to the server. The communication utilizes encryption protocols to protect data privacy.

[0565] Step 3:

[0566] The server processes the received audio data through a speech recognition engine to identify the language used in the conversation and further analyzes the intonation to estimate the speaker's regional background.

[0567] Step 4:

[0568] The server uses generative models based on language and intonation analysis to estimate cultural background and generate a cultural profile. This profile includes region-specific communication styles and values.

[0569] Step 5:

[0570] The server uses this cultural profile to translate the user's statements into language appropriate to the recipient's culture. The translated content is then adjusted to ensure that it is conveyed as intended to the recipient.

[0571] Step 6:

[0572] The emotion engine installed on the server analyzes the tone, tempo, and volume of the voice data to evaluate the user's emotional state in real time.

[0573] Step 7:

[0574] The server uses the emotional state evaluation obtained from the emotion engine to inform the translation, and also sends emotionally appropriate feedback and warnings to the user via voice as needed.

[0575] Step 8:

[0576] The device receives the converted translation data sent from the server and plays it back through the user's earphones.

[0577] Step 9:

[0578] Users receive feedback from the server as needed to adjust their conversations and facilitate smooth cross-cultural communication.

[0579] (Example 2)

[0580] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0581] Communication between individuals with different cultural backgrounds presents challenges, including the frequent occurrence of cultural misunderstandings and emotional discrepancies. Traditional interpretation systems are limited to pure language translation, making it difficult to adequately consider cultural backgrounds and emotions in communication. Therefore, new technologies are needed to enable smooth intercultural dialogue.

[0582] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0583] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative AI model based on the analysis results, and means for recognizing the emotional state from the voice using emotion analysis means and evaluating stress levels and satisfaction levels. This enables interpretation that takes cultural background and emotions into account, as well as individually tailored feedback.

[0584] "Means of collecting voice from users" refers to a function that uses a voice input device to acquire voices spoken by users in digital format.

[0585] "Means of analyzing audio data to identify language and intonation" refers to a function that uses speech analysis technology to identify the language used, intonation, and emphasis of speech from collected audio data.

[0586] "Methods for estimating cultural background using generative AI models" refers to a function that uses machine learning-based algorithms to infer the cultural context of a speaker from the content of audio data.

[0587] "Emotional analysis means" refers to technology that analyzes the characteristics of audio data to identify the speaker's emotional state, and to the function of scrutinizing the intensity and type of that emotion.

[0588] "Means of providing personalized feedback" refers to features that provide users with individually tailored advice and information based on analyzed cultural and emotional data.

[0589] "Means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a function that identifies misunderstandings and inappropriate cultural expressions hidden in the content and context of a conversation and informs the user of them.

[0590] This invention is a system for enabling smooth intercultural communication and includes the functions of voice collection, analysis, translation, and emotion recognition. Specific embodiments are shown below.

[0591] The user's voice is collected using the voice input device built into the device. For example, the microphone on a smartphone can be used. The collected voice data is sent to the server using a secure protocol.

[0592] The server uses the Google Cloud Speech-to-Text API or similar speech analysis services to identify language and intonation. Based on this analysis, a generative AI model is used to estimate the cultural background of the conversation. Using a general-purpose language model is effective for this purpose.

[0593] Furthermore, the server uses emotion analysis tools to analyze elements such as tone, tempo, and volume from the audio data to recognize the speaker's emotional state. For this purpose, speech analysis libraries and tools can be used.

[0594] Based on the analysis results, the server translates the user's statements into a culturally appropriate format. The translated content is sent to the terminal to support the conversation between the user and the other party.

[0595] As a concrete example, consider a scenario where a Japanese-speaking user interacts with a Spanish-speaking person. The user's statement, "Is that information interesting?", is translated into "Do you have a strong interest?" based on cultural background and sentiment analysis, and then communicated to the other person. Furthermore, if sentiment analysis determines that the user is experiencing stress, the server provides feedback encouraging them to shift to a more relaxed topic.

[0596] To effectively utilize the generative AI model, input should be in the format of an example prompt such as, "Explain how to analyze the user's voice data, taking into account cultural background and emotions, to produce an appropriate translation." Using this prompt makes it easier for the system to generate accurate translations and feedback.

[0597] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0598] Step 1:

[0599] The device collects the user's voice using earphones or a smartphone. The voice is converted to a digital format and encoded for transmission. The input is the user's speech, and the output is digital audio data. This is then prepared for subsequent server processing.

[0600] Step 2:

[0601] The terminal sends the collected digital audio data to the server. The data is transmitted securely using a secure protocol (e.g., HTTPS). The input is the audio data obtained in step 1, and the output is the same data received on the server side.

[0602] Step 3:

[0603] The server analyzes the received audio data to identify the language and intonation. Specifically, it uses a speech analysis API. The input is audio data, and the output is the language type and intonation pattern. Based on these results, the following processing is performed.

[0604] Step 4:

[0605] The server uses a generative AI model to estimate the cultural background based on the analysis results. The AI ​​model takes language and intonation data as input and outputs the speaker's cultural background. This process provides an appropriate context for the conversation.

[0606] Step 5:

[0607] The server processes audio data for sentiment analysis. It analyzes the tone and tempo of the voice and evaluates the emotional state. The input is audio data, and the output is an evaluation value of the emotional state (e.g., stress level or satisfaction level). This information influences the interpretation process.

[0608] Step 6:

[0609] The server translates and interprets the user's utterances, taking into account cultural background and sentiment analysis results. The appropriate expression is selected using the output of the generated AI model. The input is the user's original utterance, and the output is the translated utterance.

[0610] Step 7:

[0611] The server sends the translated utterances and feedback to the terminal. The terminal presents these to the user to aid understanding. The input is the translated utterances and feedback, and the output is what is displayed to the user.

[0612] (Application Example 2)

[0613] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0614] In intercultural communication, misunderstandings can arise not only from differences in language and cultural background, but also from subtle differences in emotional nuances. Furthermore, in multilingual environments such as movies and events, the lack of subtitles as visual information makes it difficult to accurately understand the emotions and nuances of the situation. To address these issues, a real-time interpretation system that incorporates sentiment analysis, in addition to language translation, is required.

[0615] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0616] In this invention, the server includes means for collecting speech; means for analyzing the collected speech information to identify language and intonation; means for estimating the cultural background using a generative model based on the analysis results; means for transforming speech and performing interpretation according to the estimated cultural background; means for detecting cultural taboos and risks of misunderstanding and issuing warnings to the user; means for evaluating the emotional state of the speech using sentiment analysis and reflecting the results in the interpretation; and means for generating multilingual subtitles and providing them to the user as visual information. This enables emotionally rich communication that transcends language barriers.

[0617] "Means for collecting sound" refers to a device that inputs sound data, converts it into a digital format, and supplies it to the next analysis process.

[0618] A "means for identifying language and intonation" refers to a device that has the function of analyzing and extracting the language used, as well as the intonation and tone of speech, from audio data.

[0619] "Methods for estimating cultural background using generative models" are devices that predict a speaker's cultural background by utilizing databases and algorithms based on analyzed language and intonation information.

[0620] A "means of transforming and interpreting speech" is a device that, based on an estimated cultural background, translates the original speech into appropriate linguistic expressions and mediates communication between cultures.

[0621] "A means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a device that identifies potential cultural friction points and misunderstandings that may arise during the interpretation process and informs users of them.

[0622] "A means of evaluating the emotional state of speech using emotion analysis and reflecting the results in interpretation" refers to a device that analyzes emotional information from speech and incorporates that information into the interpretation process.

[0623] "A means of generating multilingual subtitles and providing them to users as visual information" refers to a device that generates subtitles translated in real time based on audio data and presents them visually to the user.

[0624] This invention is an interpretation system designed to support smooth intercultural communication. The system is built on a cloud-based platform that runs on various devices.

[0625] System Configuration

[0626] Audio collection:

[0627] The device collects audio using microphones built into smart glasses or head-mounted displays. At this stage, the user's speech and surrounding sounds are converted into digital signals in real time and sent to a cloud server.

[0628] Voice analysis:

[0629] The server uses speech recognition engines such as Google Cloud Speech-to-Text to analyze and extract language and intonation from the input audio. This information is then used to estimate the cultural background in more detail.

[0630] Estimation of cultural background:

[0631] A generative AI model is used to estimate cultural background from the analyzed linguistic information. This estimates the speaker's cultural characteristics, and based on this, instructions are created to convert the information into appropriate linguistic expressions.

[0632] Sentiment analysis and translation:

[0633] Emotion analysis engines such as the Azure Emotion API extract emotional information from speech, and the results are then translated into the appropriate language using translation services such as the DeepL API. During this process, emotional nuances are also reflected in the selected language.

[0634] Providing visual information:

[0635] Real-time generated multilingual subtitles are displayed as visual information on a screen such as smart glasses. This display technology makes it easier for users to understand not only the language translation but also the nuances of culture and emotion.

[0636] Specific example

[0637] For example, consider a scenario at an international film festival where a film requiring multilingual support is screened. This system analyzes the film's dialogue in real time and provides translated subtitles based on cultural context and emotions. This allows audiences with different language backgrounds to experience the same emotional impact.

[0638] Example of a prompt

[0639] "Please show how to appropriately express in Japanese a scene in a film submitted to an international film festival where the protagonist is emotionally overwhelmed."

[0640] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0641] Step 1: Collect audio data

[0642] The device uses a microphone built into the smart glasses to collect user and ambient sounds. The collected analog audio is converted into a digital signal and sent to a cloud server. The input is the user's voice and ambient sounds, and the output is the digitized audio data.

[0643] Step 2: Speech Recognition and Analysis

[0644] The server uses Google Cloud Speech-to-Text to convert the transmitted audio data into text format. Next, it analyzes the language and intonation of the speech to generate foundational data for estimating the cultural background. The input is digitized audio data, and the output is the spoken content in text format, along with corresponding intonation information.

[0645] Step 3: Estimating the Cultural Background

[0646] The server uses a generative AI model to estimate cultural background from analyzed linguistic information. In this process, an estimation algorithm operates based on the generated data, extracting cultural context relevant to the country, region, or situation. The input is text and intonation data, and the output is estimated information about the cultural background.

[0647] Step 4: Conducting sentiment analysis

[0648] The server uses the Azure Emotion API to analyze and extract emotional information from audio data. This analysis generates information based on the speaker's emotional state and tone. The input is audio data, and the output is emotional information, specifically an evaluation of the speaker's stress level and satisfaction level.

[0649] Step 5: Perform translation and interpretation.

[0650] The server uses the DeepL API to translate utterances into appropriate language, taking into account cultural context and emotional information. This step specifically adjusts the translation to reflect emotions and cultural nuances. The input is cultural context and emotional information, and the output is the translated result expressed in the desired language.

[0651] Step 6: Generate and display multilingual subtitles

[0652] The device displays the translation results as visual information on smart glasses. At this stage, the information is presented as real-time subtitles, providing the user with an interactive experience that transcends language and cultural barriers. The input is translated speech data, and the output is multilingual subtitles displayed to the user.

[0653] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0654] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0655] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0656] [Fourth Embodiment]

[0657] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0658] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0659] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0660] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0661] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0662] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0663] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0664] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0665] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0666] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0667] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0668] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0669] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0670] This invention includes a series of devices and processes necessary to implement an interpretation system that takes cultural background into account. A device worn by the user, such as an earphone or smartphone, collects audio data. This allows the conversation between the user and the other party to be monitored in real time and transmitted to a server.

[0671] The server performs speech recognition and analysis on the received audio data to identify the language and intonation of the conversation. Using generative models based on this identified information, it estimates the other party's cultural background. This allows the server to transform the user's statements into appropriate expressions that take into account the other party's communication style and values.

[0672] The server performs a translation process according to the cultural background, sends the converted audio information back to the user's device, and outputs it through the earphones. This allows the user to receive messages in a structure appropriate to the other person's culture.

[0673] Furthermore, the server constantly monitors the context of the conversation and, if it detects a risk of cultural taboos or misunderstandings, sends an alert to the user via voice warning. This alert includes appropriate alternative expressions and suggestions for changing the topic to help the user avoid misunderstandings.

[0674] One concrete example is a conversation between a Japanese-speaking user and a French-speaking partner where a topic related to a specific expression in French culture is discussed, and it is then translated into something that is commonly discussed in Japanese culture. For instance, if a topic of a particular holiday or custom popular in France comes up, it could be changed to a discussion of a similar holiday or custom in Japan, thereby promoting cross-cultural understanding.

[0675] This system will facilitate smoother and more considerate international and intercultural communication, enabling dialogue with minimal misunderstandings.

[0676] The following describes the processing flow.

[0677] Step 1:

[0678] The device collects the conversation audio between the user and the other party in real time using the microphone in the earphone. The collected audio data is then de-noised and prepared for the next processing step as clear audio data.

[0679] Step 2:

[0680] The terminal sends the pre-processed audio data to the server via a network such as the internet. The network connection utilizes encryption protocols for data protection.

[0681] Step 3:

[0682] The server places the received audio data into a queue for analysis. During this process, speech recognition technology is used to identify the language being used in the audio data.

[0683] Step 4:

[0684] The server analyzes the intonation of the audio data to estimate the speaker's regional or cultural background. It identifies geographical features based on tone, rhythm, and accent.

[0685] Step 5:

[0686] The server uses a generative model to generate cultural profiles based on identified cultural elements, such as politeness and directness.

[0687] Step 6:

[0688] Based on the generated cultural profile, the server translates the user's utterances into language that is easily understood by the recipient. This translation is adjusted to match the selected context.

[0689] Step 7:

[0690] The server sends the converted language data back to the terminal. The terminal converts this data into an audio signal and transmits it to the user in real time via earphones.

[0691] Step 8:

[0692] The server performs continuous response analysis to detect potential cultural taboos and misunderstandings during conversations. If a risk is detected, the server sends an audio warning to the user.

[0693] Step 9:

[0694] The server generates alternative expressions and suggestions for new topics based on the risk of misunderstanding, and provides these to the user along with alerts. This helps the user to appropriately adjust the conversation.

[0695] (Example 1)

[0696] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0697] In today's global society, opportunities for people with different languages ​​and cultural backgrounds to communicate are increasing. In these situations, not only language barriers but also cultural misunderstandings and taboos can become major obstacles. Traditional interpretation systems focus on language conversion, but they often lack sufficient consideration and adaptation to cultural backgrounds, potentially hindering smooth communication and leading to misunderstandings.

[0698] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0699] In this invention, the server includes means for collecting auditory information from the user, means for analyzing the collected auditory information to identify linguistic features and intonation, and means for estimating the cultural background using a generation algorithm based on the analysis results. This enables smooth intercultural communication by providing appropriate interpretation and warnings while considering the cultural context.

[0700] "User" refers to a person who uses the system to collect and interpret audio data.

[0701] "Auditory information" refers to data that includes audio data and language information collected from users.

[0702] "Linguistic features" refer to the types of language, unique expressions, and vocabulary patterns extracted from audio data.

[0703] "Intonation" refers to elements that describe the characteristics of speech in language, such as intonation, volume, and rhythm.

[0704] A "generative algorithm" refers to a set of computational procedures or models used to generate or transform information.

[0705] "Cultural background" refers to the values, beliefs, and norms that are generally shared within a particular culture or society.

[0706] "Interpretation" refers to the act of facilitating mutual understanding by appropriately conveying content expressed in one language into another language.

[0707] A "cultural taboo" refers to actions, expressions, or topics that are considered to be avoided in a particular culture.

[0708] The "risk of misunderstanding" refers to the possibility that the intended meaning in communication may differ from the meaning that is received.

[0709] A "warning" refers to information or a message issued to draw attention to a user.

[0710] "Communication style" refers to the methods and styles of communication commonly used in a particular culture or society.

[0711] This invention provides a multi-functional interpretation system to facilitate intercultural communication. Users collect audio using earphones or a smartphone. Specifically, a wireless communication device (e.g., wireless earphones) is used as the earphone to collect audio data and transmit it to the terminal in real time.

[0712] The terminal has communication capabilities to send received audio to the server. The server converts the audio data into text using speech recognition software (e.g., a speech-to-text algorithm). The server analyzes linguistic features and intonation and uses a generation algorithm to estimate the other party's cultural background. This analysis generates a cultural profile using a general computational model.

[0713] Based on the estimated cultural background, the server translates the user's utterances into expressions appropriate to the target language's culture. This process utilizes a generative AI model, for example, using prompts such as, "Please translate this expression from French culture into a form suitable for Japanese culture."

[0714] The converted data is sent back from the server to the terminal. The terminal outputs the converted audio to the user through earphones. The user can communicate smoothly with the other party through this converted audio. Furthermore, the server monitors the context of the conversation and issues a warning to the user if it detects a topic that could be misunderstood. Specifically, an alert is generated that says, "This topic is culturally sensitive. Please choose a different topic."

[0715] This system enables users to have smooth and less misunderstanding-free conversations with people from different cultural backgrounds.

[0716] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0717] Step 1:

[0718] The user collects audio using earphones and a smartphone. The earphones capture ambient sound through a microphone and transmit it to the smartphone. Here, the input is ambient sound, and the output is digital audio data transferred to the smartphone.

[0719] Step 2:

[0720] The device compresses the collected audio data and transmits it to the server via the internet. The input is digital audio data on the smartphone, and the output is compressed audio data securely transmitted over the internet. Specifically, data compression software runs within the device.

[0721] Step 3:

[0722] The server decompresses the received compressed audio data and converts it to text using a speech recognition algorithm. The input is the compressed audio data received by the server, and the output is the text data generated after speech recognition. Advanced speech recognition software is used in this step.

[0723] Step 4:

[0724] The server uses speech analysis software to identify linguistic features and intonation from text data and sends the analysis results to a generating AI model. The input is text data generated by speech recognition, and the output is the analyzed linguistic features and intonation information. The server generates a prompt sentence, which the generating AI model then uses to operate.

[0725] Step 5:

[0726] The generative AI model estimates cultural background based on analyzed linguistic features and intonation information. The input is the analysis results sent from the server, and the output is the estimated cultural background information. Specifically, the AI ​​processes prompts such as, "Please convert this into a format suitable for Japanese culture."

[0727] Step 6:

[0728] The server converts the user's utterances into culturally appropriate expressions based on the cultural background information it receives. The input is the user's original utterance data and cultural background information, and the output is the modified utterance data. The server runs software responsible for adjusting the utterances.

[0729] Step 7:

[0730] The server converts the transformed speech data back into speech using speech synthesis technology and sends it to the terminal. The input is the user's modified text data, and the output is the data converted into a speech format.

[0731] Step 8:

[0732] The terminal sends the received audio data to the earphones and plays it back to the user. The input is the converted audio data sent from the server, and the output is the audio the user hears through the earphones. Specifically, audio playback software runs on the terminal.

[0733] Step 9:

[0734] The server monitors the conversation in real time and sends an alert to the user if it detects a risk of cultural taboos or misunderstandings. The input is continuously updated conversation data, and the output is alert information. At this stage, the warning system is active, and an audible alert is delivered through the terminal.

[0735] (Application Example 1)

[0736] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0737] In modern society, intercultural communication is a crucial element, but misunderstandings and inappropriate communication can occur due to differences in cultural backgrounds and values. Especially in today's world, where interaction with individuals from different cultural backgrounds is frequent within families and in daily life, there is a need for effective means to overcome these cultural barriers. Traditional interpretation systems often limit themselves to language translation, failing to adjust communication to cultural contexts.

[0738] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0739] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative model, means for adjusting cross-cultural communication and translating the user's statements, means for detecting the risk of cultural taboos and misunderstandings and issuing warnings to the user, and means for supporting dialogue between individuals with different cultural backgrounds within the household. This enables smooth intercultural communication and effectively reduces misunderstandings and communication barriers caused by cultural differences.

[0740] A "user" is an individual who uses a voice translation system to communicate across cultures.

[0741] "Voice data" refers to information obtained by collecting the voice spoken by a user using digital technology and converting it into an analyzable format.

[0742] "Analysis" is the process of identifying the linguistic characteristics and intonation of collected audio data and deriving results from that information.

[0743] A "generative model" is an algorithm that uses artificial intelligence to produce diverse outputs based on specific conditions and input information.

[0744] "Cultural background" refers to the totality of values ​​and communication styles formed from an individual's culture of origin and social environment.

[0745] "Intercultural communication" refers to the transmission of information and communication that takes place between people with different cultural backgrounds.

[0746] A "cultural taboo" refers to an action, expression, or topic that is unacceptable in a particular culture or society.

[0747] The system for implementing this invention primarily uses voice data as a medium between the user and the server. The user communicates via voice, and this voice data is collected using the microphone of a device. This device includes smartphones and home robots. The collected voice data is transmitted to the server in real time.

[0748] The server uses speech recognition software to convert audio data into text data. Technologies such as the Google Cloud Speech-to-Text API are used for this process. The server then analyzes the converted text data to identify the linguistic characteristics and intonation of the speech. Based on the analysis results, a generative AI model such as OpenAI's GPT-4 is used to estimate the cultural background of the other party.

[0749] Based on this estimation, the server transforms the user's speech to fit the identified cultural background, adjusting it to minimize the risk of cultural taboos and misunderstandings. The transformed audio is then provided to the user again as audio data and output through the user's device.

[0750] This system facilitates smooth intercultural communication within the home, avoiding taboos and misunderstandings specific to certain cultures. For example, if a foreign partner visiting a Japanese home mentions a holiday in their home country, the robot automatically translates it to a similar holiday in the host country. In this way, cultural understanding is deepened, and a better communication experience is provided.

[0751] An example of a prompt is: "Please translate the current conversation to align with the values ​​of another culture, taking cultural context into consideration. While maintaining the context of the conversation, replace it with appropriate similar holidays or topics in Japanese culture." Through this prompt, the server enables flexible communication that takes cultural context into account.

[0752] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0753] Step 1:

[0754] The user inputs voice through the device's microphone. This input is natural conversational audio data. The device collects this audio data in real time, converts it to a digital format, and sends it to a server. This conversion process includes pre-processing such as audio compression and noise reduction.

[0755] Step 2:

[0756] The server converts received audio data into text data using the Google Cloud Speech-to-Text API. Audio data is input, and text data is output. During the conversion process, phoneme analysis and natural language processing techniques are used to identify language characteristics and intonation.

[0757] Step 3:

[0758] The server uses OpenAI's GPT-4 to generate an AI model that takes the converted text data as input and estimates the other party's cultural background. This estimation extracts culture-related features from words and phrases within the text and generates an estimation result based on the prompt sentence. The output is the estimated cultural profile.

[0759] Step 4:

[0760] The server takes the estimated cultural profile into consideration and appropriately transforms the user's utterances. The input is the original text data and cultural profile, and the output is the newly text of the culturally sensitive utterance. This transformation process includes adjustments to avoid cultural taboos and misunderstandings.

[0761] Step 5:

[0762] The server generates the adjusted text data and converts it back into speech using speech synthesis technology. The generated speech data is then sent from the server to the user's terminal. A speech synthesis algorithm is used here, taking into account sound quality and fluency.

[0763] Step 6:

[0764] The user's device outputs the received audio through its speaker. This allows the user to receive culturally context-adjusted messages in audio format. The audio output also includes volume adjustment and sound quality optimization.

[0765] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0766] This invention combines an interpretation system that takes cultural background into account with an emotion engine. The system incorporates a device that collects audio from the user; the terminal collects the conversation between the user and the other party in real time via earphones or a smartphone. The collected audio data is transmitted to a server for analysis.

[0767] After acquiring audio data, the server performs language recognition and intonation analysis to identify the language and regional characteristics being used. Based on this information, it uses a generative model to estimate the cultural background of the conversation. Then, based on the estimated culture, the server translates the user's statements and provides interpretation using appropriate linguistic expressions. This facilitates smoother intercultural communication.

[0768] In addition, the server uses an emotion engine to recognize emotions from the user's voice. This engine analyzes the user's emotional state from factors such as tone, tempo, and volume, and evaluates stress levels and satisfaction levels. The results of this emotion analysis are reflected in the translation and warning system in real time, providing more personalized feedback.

[0769] As a concrete example, consider a conversation between a Japanese-speaking user and a Spanish-speaking person. In this case, the user's question, "Is that information interesting?", is translated to "Does it pique your interest?", taking into account the other person's cultural background and emotions, thus conveying the emotional nuance as well. Furthermore, if the emotion engine detects that the user is stressed, the server sends an alert to encourage a shift to a calmer topic to avoid excessive conversation. This kind of feedback improves both the accuracy of the translation and the user experience.

[0770] This system supports intercultural exchange by creating a more sophisticated and considerate communication environment that takes emotions and cultural backgrounds into account.

[0771] The following describes the processing flow.

[0772] Step 1:

[0773] The device collects the conversation between the user and the other party as audio data through the microphone in the earphone. The collected audio data is temporarily stored on the device.

[0774] Step 2:

[0775] The device sends the collected voice data to the server. The communication utilizes encryption protocols to protect data privacy.

[0776] Step 3:

[0777] The server processes the received audio data through a speech recognition engine to identify the language used in the conversation and further analyzes the intonation to estimate the speaker's regional background.

[0778] Step 4:

[0779] The server uses generative models based on language and intonation analysis to estimate cultural background and generate a cultural profile. This profile includes region-specific communication styles and values.

[0780] Step 5:

[0781] The server uses this cultural profile to translate the user's statements into language appropriate to the recipient's culture. The translated content is then adjusted to ensure that it is conveyed as intended to the recipient.

[0782] Step 6:

[0783] The emotion engine installed on the server analyzes the tone, tempo, and volume of the voice data to evaluate the user's emotional state in real time.

[0784] Step 7:

[0785] The server uses the emotional state evaluation obtained from the emotion engine to inform the translation, and also sends emotionally appropriate feedback and warnings to the user via voice as needed.

[0786] Step 8:

[0787] The device receives the converted translation data sent from the server and plays it back through the user's earphones.

[0788] Step 9:

[0789] Users receive feedback from the server as needed to adjust their conversations and facilitate smooth cross-cultural communication.

[0790] (Example 2)

[0791] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0792] Communication between individuals with different cultural backgrounds presents challenges, including the frequent occurrence of cultural misunderstandings and emotional discrepancies. Traditional interpretation systems are limited to pure language translation, making it difficult to adequately consider cultural backgrounds and emotions in communication. Therefore, new technologies are needed to enable smooth intercultural dialogue.

[0793] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0794] In this invention, the server includes means for collecting voice from the user, means for analyzing the collected voice data to identify language and intonation, means for estimating the cultural background using a generative AI model based on the analysis results, and means for recognizing the emotional state from the voice using emotion analysis means and evaluating stress levels and satisfaction levels. This enables interpretation that takes cultural background and emotions into account, as well as individually tailored feedback.

[0795] "Means of collecting voice from users" refers to a function that uses a voice input device to acquire voices spoken by users in digital format.

[0796] "Means of analyzing audio data to identify language and intonation" refers to a function that uses speech analysis technology to identify the language used, intonation, and emphasis of speech from collected audio data.

[0797] "Methods for estimating cultural background using generative AI models" refers to a function that uses machine learning-based algorithms to infer the cultural context of a speaker from the content of audio data.

[0798] "Emotional analysis means" refers to technology that analyzes the characteristics of audio data to identify the speaker's emotional state, and to the function of scrutinizing the intensity and type of that emotion.

[0799] "Means of providing personalized feedback" refers to features that provide users with individually tailored advice and information based on analyzed cultural and emotional data.

[0800] "Means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a function that identifies misunderstandings and inappropriate cultural expressions hidden in the content and context of a conversation and informs the user of them.

[0801] This invention is a system for enabling smooth intercultural communication and includes the functions of voice collection, analysis, translation, and emotion recognition. Specific embodiments are shown below.

[0802] The user's voice is collected using the voice input device built into the device. For example, the microphone on a smartphone can be used. The collected voice data is sent to the server using a secure protocol.

[0803] The server uses the Google Cloud Speech-to-Text API or similar speech analysis services to identify language and intonation. Based on this analysis, a generative AI model is used to estimate the cultural background of the conversation. Using a general-purpose language model is effective for this purpose.

[0804] Furthermore, the server uses emotion analysis tools to analyze elements such as tone, tempo, and volume from the audio data to recognize the speaker's emotional state. For this purpose, speech analysis libraries and tools can be used.

[0805] Based on the analysis results, the server translates the user's statements into a culturally appropriate format. The translated content is sent to the terminal to support the conversation between the user and the other party.

[0806] As a concrete example, consider a scenario where a Japanese-speaking user interacts with a Spanish-speaking person. The user's statement, "Is that information interesting?", is translated into "Do you have a strong interest?" based on cultural background and sentiment analysis, and then communicated to the other person. Furthermore, if sentiment analysis determines that the user is experiencing stress, the server provides feedback encouraging them to shift to a more relaxed topic.

[0807] To effectively utilize the generative AI model, input should be in the format of an example prompt such as, "Explain how to analyze the user's voice data, taking into account cultural background and emotions, to produce an appropriate translation." Using this prompt makes it easier for the system to generate accurate translations and feedback.

[0808] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0809] Step 1:

[0810] The device collects the user's voice using earphones or a smartphone. The voice is converted to a digital format and encoded for transmission. The input is the user's speech, and the output is digital audio data. This is then prepared for subsequent server processing.

[0811] Step 2:

[0812] The terminal sends the collected digital audio data to the server. The data is transmitted securely using a secure protocol (e.g., HTTPS). The input is the audio data obtained in step 1, and the output is the same data received on the server side.

[0813] Step 3:

[0814] The server analyzes the received audio data to identify the language and intonation. Specifically, it uses a speech analysis API. The input is audio data, and the output is the language type and intonation pattern. Based on these results, the following processing is performed.

[0815] Step 4:

[0816] The server uses a generative AI model to estimate the cultural background based on the analysis results. The AI ​​model takes language and intonation data as input and outputs the speaker's cultural background. This process provides an appropriate context for the conversation.

[0817] Step 5:

[0818] The server processes audio data for sentiment analysis. It analyzes the tone and tempo of the voice and evaluates the emotional state. The input is audio data, and the output is an evaluation value of the emotional state (e.g., stress level or satisfaction level). This information influences the interpretation process.

[0819] Step 6:

[0820] The server translates and interprets the user's utterances, taking into account cultural background and sentiment analysis results. The appropriate expression is selected using the output of the generated AI model. The input is the user's original utterance, and the output is the translated utterance.

[0821] Step 7:

[0822] The server sends the translated utterances and feedback to the terminal. The terminal presents these to the user to aid understanding. The input is the translated utterances and feedback, and the output is what is displayed to the user.

[0823] (Application Example 2)

[0824] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0825] In intercultural communication, misunderstandings can arise not only from differences in language and cultural background, but also from subtle differences in emotional nuances. Furthermore, in multilingual environments such as movies and events, the lack of subtitles as visual information makes it difficult to accurately understand the emotions and nuances of the situation. To address these issues, a real-time interpretation system that incorporates sentiment analysis, in addition to language translation, is required.

[0826] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0827] In this invention, the server includes means for collecting speech; means for analyzing the collected speech information to identify language and intonation; means for estimating the cultural background using a generative model based on the analysis results; means for transforming speech and performing interpretation according to the estimated cultural background; means for detecting cultural taboos and risks of misunderstanding and issuing warnings to the user; means for evaluating the emotional state of the speech using sentiment analysis and reflecting the results in the interpretation; and means for generating multilingual subtitles and providing them to the user as visual information. This enables emotionally rich communication that transcends language barriers.

[0828] "Means for collecting sound" refers to a device that inputs sound data, converts it into a digital format, and supplies it to the next analysis process.

[0829] A "means for identifying language and intonation" refers to a device that has the function of analyzing and extracting the language used, as well as the intonation and tone of speech, from audio data.

[0830] "Methods for estimating cultural background using generative models" are devices that predict a speaker's cultural background by utilizing databases and algorithms based on analyzed language and intonation information.

[0831] A "means of transforming and interpreting speech" is a device that, based on an estimated cultural background, translates the original speech into appropriate linguistic expressions and mediates communication between cultures.

[0832] "A means of detecting the risk of cultural taboos and misunderstandings and warning users" refers to a device that identifies potential cultural friction points and misunderstandings that may arise during the interpretation process and informs users of them.

[0833] "A means of evaluating the emotional state of speech using emotion analysis and reflecting the results in interpretation" refers to a device that analyzes emotional information from speech and incorporates that information into the interpretation process.

[0834] "A means of generating multilingual subtitles and providing them to users as visual information" refers to a device that generates subtitles translated in real time based on audio data and presents them visually to the user.

[0835] This invention is an interpretation system designed to support smooth intercultural communication. The system is built on a cloud-based platform that runs on various devices.

[0836] System Configuration

[0837] Audio collection:

[0838] The device collects audio using microphones built into smart glasses or head-mounted displays. At this stage, the user's speech and surrounding sounds are converted into digital signals in real time and sent to a cloud server.

[0839] Voice analysis:

[0840] The server uses speech recognition engines such as Google Cloud Speech-to-Text to analyze and extract language and intonation from the input audio. This information is then used to estimate the cultural background in more detail.

[0841] Estimation of cultural background:

[0842] A generative AI model is used to estimate cultural background from the analyzed linguistic information. This estimates the speaker's cultural characteristics, and based on this, instructions are created to convert the information into appropriate linguistic expressions.

[0843] Sentiment analysis and translation:

[0844] Emotion analysis engines such as the Azure Emotion API extract emotional information from speech, and the results are then translated into the appropriate language using translation services such as the DeepL API. During this process, emotional nuances are also reflected in the selected language.

[0845] Providing visual information:

[0846] Real-time generated multilingual subtitles are displayed as visual information on a screen such as smart glasses. This display technology makes it easier for users to understand not only the language translation but also the nuances of culture and emotion.

[0847] Specific example

[0848] For example, consider a scenario at an international film festival where a film requiring multilingual support is screened. This system analyzes the film's dialogue in real time and provides translated subtitles based on cultural context and emotions. This allows audiences with different language backgrounds to experience the same emotional impact.

[0849] Example of a prompt

[0850] "Please show how to appropriately express in Japanese a scene in a film submitted to an international film festival where the protagonist is emotionally overwhelmed."

[0851] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0852] Step 1: Collect audio data

[0853] The device uses a microphone built into the smart glasses to collect user and ambient sounds. The collected analog audio is converted into a digital signal and sent to a cloud server. The input is the user's voice and ambient sounds, and the output is the digitized audio data.

[0854] Step 2: Speech Recognition and Analysis

[0855] The server uses Google Cloud Speech-to-Text to convert the transmitted audio data into text format. Next, it analyzes the language and intonation of the speech to generate foundational data for estimating the cultural background. The input is digitized audio data, and the output is the spoken content in text format, along with corresponding intonation information.

[0856] Step 3: Estimating the Cultural Background

[0857] The server uses a generative AI model to estimate cultural background from analyzed linguistic information. In this process, an estimation algorithm operates based on the generated data, extracting cultural context relevant to the country, region, or situation. The input is text and intonation data, and the output is estimated information about the cultural background.

[0858] Step 4: Conducting sentiment analysis

[0859] The server uses the Azure Emotion API to analyze and extract emotional information from audio data. This analysis generates information based on the speaker's emotional state and tone. The input is audio data, and the output is emotional information, specifically an evaluation of the speaker's stress level and satisfaction level.

[0860] Step 5: Perform translation and interpretation.

[0861] The server uses the DeepL API to translate utterances into appropriate language, taking into account cultural context and emotional information. This step specifically adjusts the translation to reflect emotions and cultural nuances. The input is cultural context and emotional information, and the output is the translated result expressed in the desired language.

[0862] Step 6: Generate and display multilingual subtitles

[0863] The device displays the translation results as visual information on smart glasses. At this stage, the information is presented as real-time subtitles, providing the user with an interactive experience that transcends language and cultural barriers. The input is translated speech data, and the output is multilingual subtitles displayed to the user.

[0864] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0865] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0866] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0867] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0868] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0869] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0870] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0871] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0872] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0873] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0874] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0875] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0876] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0877] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0878] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0879] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0880] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0881] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0882] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0883] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0884] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0885] The following is further disclosed regarding the embodiments described above.

[0886] (Claim 1)

[0887] A device that collects voice from the user,

[0888] A device that analyzes collected audio data to identify language and intonation,

[0889] Based on the analysis results, a device is used to estimate the cultural background using a generative model,

[0890] A device that translates and interprets user speech according to the estimated cultural background,

[0891] A device that detects the risk of cultural taboos and misunderstandings and warns the user,

[0892] A system that includes this.

[0893] (Claim 2)

[0894] The system according to claim 1, which analyzes the other party's reaction in real time and provides feedback to the user based on the changes therein.

[0895] (Claim 3)

[0896] The system according to claim 1, which identifies communication styles based on cultural profiles generated by analysis.

[0897] "Example 1"

[0898] (Claim 1)

[0899] Means of collecting auditory information from users,

[0900] A means of analyzing collected auditory information to identify linguistic features and intonation,

[0901] Based on the analysis results, a means of estimating the cultural background using a generative algorithm,

[0902] A means of translating and interpreting the user's statements according to the estimated cultural background,

[0903] A means of detecting the risk of cultural taboos and misunderstandings and warning users,

[0904] A means of monitoring the flow of conversation and generating culturally sensitive expressions,

[0905] A system that includes this.

[0906] (Claim 2)

[0907] The system according to claim 1, which analyzes the other party's reaction in real time and provides feedback to the user based on the changes therein.

[0908] (Claim 3)

[0909] The system according to claim 1, which identifies interaction patterns based on cultural profiles generated by analysis.

[0910] "Application Example 1"

[0911] (Claim 1)

[0912] Means of collecting voice from users,

[0913] A means of analyzing collected audio data to identify language and intonation,

[0914] Based on the analysis results, a means of estimating the cultural background using a generative model,

[0915] A means of adjusting cross-cultural communication according to the estimated cultural background and translating and interpreting the user's statements,

[0916] A means to detect the risk of cultural taboos and misunderstandings and warn users,

[0917] Means to support dialogue between individuals with different cultural backgrounds within the family,

[0918] A system that includes this.

[0919] (Claim 2)

[0920] The system according to claim 1, which analyzes the other party's reaction in real time and provides feedback to the user based on the changes therein.

[0921] (Claim 3)

[0922] The system according to claim 1, which identifies communication styles based on cultural profiles generated by analysis and enhances support for home-based dialogue.

[0923] "Example 2 of combining an emotion engine"

[0924] (Claim 1)

[0925] Means of collecting voice from users,

[0926] A means of analyzing collected audio data to identify language and intonation,

[0927] Based on the analysis results, a means of estimating the cultural background using a generative AI model,

[0928] A means of translating and interpreting user statements according to the estimated cultural background,

[0929] A means of recognizing emotional states from voice using emotion analysis methods and evaluating stress levels and satisfaction levels,

[0930] A means of providing personalized feedback to users based on evaluation results,

[0931] A means to detect the risk of cultural taboos and misunderstandings and warn users,

[0932] A system that includes this.

[0933] (Claim 2)

[0934] The system according to claim 1, which analyzes the other party's reaction in real time and provides feedback to the user based on the changes therein.

[0935] (Claim 3)

[0936] The system according to claim 1, which identifies communication styles based on cultural profiles and sentiment analysis results generated by analysis.

[0937] "Application example 2 when combining with an emotional engine"

[0938] (Claim 1)

[0939] Means of collecting sound,

[0940] A means of analyzing collected audio information to identify language and intonation,

[0941] Based on the analysis results, a means of estimating the cultural background using a generative model,

[0942] A means of translating and interpreting speech according to the estimated cultural background,

[0943] A means to detect the risk of cultural taboos and misunderstandings and warn users,

[0944] A method for evaluating the emotional state of speech using emotion analysis and reflecting the results in interpretation,

[0945] A means of generating multilingual subtitles and providing them to users as visual information,

[0946] A system that includes this.

[0947] (Claim 2)

[0948] The system according to claim 1, which analyzes the other party's reactions and emotional changes in real time and provides feedback to the user based on those changes.

[0949] (Claim 3)

[0950] The system according to claim 1, which identifies communication styles based on cultural profiles and emotional information generated by analysis. [Explanation of symbols]

[0951] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of collecting voice from users, A means of analyzing collected audio data to identify language and intonation, Based on the analysis results, a means of estimating the cultural background using a generative model, A means of adjusting cross-cultural communication according to the estimated cultural background and translating and interpreting the user's statements, A means to detect the risk of cultural taboos and misunderstandings and warn users, Means to support dialogue between individuals with different cultural backgrounds within the family, A system that includes this.

2. The system according to claim 1, which analyzes the other party's reaction in real time and provides feedback to the user based on the changes therein.

3. The system according to claim 1, which identifies communication styles based on cultural profiles generated by analysis and enhances support for home-based dialogue.

Images