System

Abstract

To provide a system according to an embodiment, capable of providing sound of another language to which a vocal quality of a user is reflected in real time.SOLUTION: A system according to an embodiment comprises a sound collection unit, a sound selection unit, a sound conversion unit and a sound transmission unit. The sound collection unit collects a sound of a user. The sound selection unit selects a sound similar to that in a voice quality of the user from a sound data storage on the basis of sound data collected by the sound collection unit. The sound conversion unit converts the sound into another language by a generative AI on the basis of the sound selected by the sound selection unit. The sound transmission unit transmits the sound converted by the sound conversion unit to an earphone.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, speech converted into another language in real time does not reflect the quality of the user's voice, making it difficult to communicate while retaining individuality.

[0005] The system according to the embodiment aims to provide voice in another language that reflects the voice quality of the user in real time. [Means for solving the problem]

[0006] The system according to the embodiment includes a voice collection unit, a voice selection unit, a voice conversion unit, and a voice transmission unit. The voice collection unit collects the user's voice. The voice selection unit selects a voice that is close to the user's voice quality from the voice data storage based on the voice data collected by the voice collection unit. The voice conversion unit converts the voice selected by the voice selection unit into another language using a generation AI. The voice transmission unit transmits the voice converted by the voice conversion unit to the earphones. Effect of the Invention

[0007] The system according to the embodiment can provide voice in another language that reflects the user's voice quality in real time. [Brief description of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). Display 40A displays visible information such as text and images according to instructions from processor 46. Speaker 40B outputs voice according to instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Also, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, a home appliance, etc.) owned by a user. Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) The voice conversion system according to the embodiment of the present invention is a system that converts the user's voice into another language in real time and transmits it through earphones. This voice conversion system collects the user's voice, selects a voice similar to the user's voice quality from a voice database, converts the voice into another language using a generation AI, and transmits the converted voice to the earphones. This allows the user to overcome language barriers and freely enjoy communication while maintaining their individuality. For example, when a user says "Hello," the voice data is collected and a voice similar to the user's voice quality is selected from the voice database. Next, the generation AI converts the selected voice into "Hello," and the converted voice is transmitted to the earphones. This allows the user to speak in Japanese, but the other person will hear it in English. This allows smooth communication between different cultures and promotes international exchange. This allows the voice conversion system to convert the user's voice into another language in real time and freely enjoy communication while maintaining their individuality.

[0029] The voice conversion system according to the embodiment includes a voice collection unit, a voice selection unit, a voice conversion unit, and a voice transmission unit. The voice collection unit collects the user's voice. The user's voice includes, for example, speech characteristics and voice quality, but is not limited to such examples. The voice collection unit collects the user's voice using, for example, a microphone. The voice collection unit can also analyze the user's voice quality and speech characteristics. For example, the voice collection unit analyzes the user's voice quality based on frequency characteristics and timbre. The voice selection unit selects a voice close to the user's voice quality from the voice data storage based on the voice data collected by the voice collection unit. Voices of various voice qualities are stored in the voice data storage, and the voice selection unit selects a voice closest to the user's voice quality. For example, the voice selection unit selects a low-pitched voice for a user with a low-pitched voice quality. The voice conversion unit converts the voice selected by the voice selection unit into another language using a generation AI. The generation AI converts the voice into another language using, for example, a text generation AI (for example, LLM). The generation AI converts the voice into another language while preserving the user's voice quality and speaking characteristics. For example, when the user says "Hello," the generation AI converts the voice into "Hello." The voice transmission unit transmits the voice converted by the voice conversion unit to the earphones. The voice transmission unit transmits the converted voice to the earphones using wireless communication technology such as Bluetooth (registered trademark) or Wi-Fi. This allows the user to listen to the voice in another language in real time. This allows the voice conversion system according to the embodiment to convert the user's voice into another language in real time, allowing the user to freely enjoy communication while preserving their individuality.

[0030] The voice collecting unit collects the user's voice. The user's voice includes, for example, speech characteristics and voice quality, but is not limited to such examples. The voice collecting unit collects the user's voice using, for example, a microphone. Specifically, a high-sensitivity microphone can be used to clearly collect the user's voice. The microphone has a noise canceling function, which removes ambient noise and accurately collects only the user's voice. The voice collecting unit can also analyze the user's voice quality and speech characteristics. For example, the voice collecting unit analyzes the user's voice quality based on frequency characteristics and timbre. Specifically, the voice signal is digitized and the frequency spectrum is analyzed to extract the characteristics of the user's voice. This allows the voice collecting unit to grasp the characteristics of the user's voice, such as the pitch, timbre, and rhythm, in detail. Furthermore, the voice collecting unit can also analyze the user's speech pattern and accent. For example, it is possible to analyze how the user pronounces a specific word and what kind of rhythm he or she speaks with, and improve the accuracy of voice conversion based on this information. This allows the voice collecting unit to collect and analyze the user's voice with high accuracy.

[0031] The voice selection unit selects a voice close to the user's voice quality from the voice data storage based on the voice data collected by the voice collection unit. Voices of various voice qualities are stored in the voice data storage, and the voice selection unit selects a voice closest to the user's voice quality. Specifically, the voice selection unit analyzes the voice data stored in the voice data storage and searches for a voice having characteristics matching the user's voice quality. For example, the voice selection unit analyzes the frequency characteristics and timbre of the voice data stored in the voice data storage and identifies a voice matching the user's voice quality. Furthermore, the voice selection unit selects a voice taking into consideration the characteristics of the user's speaking style. For example, if the user has a low-pitched voice quality, the voice selection unit selects a low-pitched voice. Also, if the user speaks with a specific accent or rhythm, the voice selection unit selects a voice having these characteristics. In this way, the voice selection unit can select a voice closest to the characteristics of the user's voice quality and speaking style, and improve the accuracy of voice conversion.

[0032] The speech conversion unit converts the speech selected by the speech selection unit into another language using the generation AI. The generation AI converts the speech into another language using, for example, a text generation AI (for example, LLM). Specifically, the generation AI converts the speech data into text data, and generates a speech in another language based on the text data. The generation AI converts the speech into another language while preserving the user's voice quality and speaking characteristics. For example, when the user says "Hello," the generation AI converts the speech into "Hello." The generation AI analyzes the speech data and extracts the user's voice quality and speaking characteristics. Next, the generation AI generates a speech in another language based on the extracted characteristics. This allows the generation AI to generate a speech in another language while preserving the user's individuality. Furthermore, the generation AI can continuously learn to improve the accuracy of the speech conversion. For example, the generation AI updates the speech conversion model based on the user's feedback, and improves the conversion accuracy. This allows the speech conversion unit to convert the speech into another language with high accuracy while preserving the user's voice quality and speaking characteristics.

[0033] The audio transmission unit transmits the audio converted by the audio conversion unit to the earphone. The audio transmission unit transmits the converted audio to the earphone using a wireless communication technology such as Bluetooth or Wi-Fi. Specifically, the audio transmission unit transmits the converted audio data to the earphone via the wireless communication module. When Bluetooth is used, the audio transmission unit compresses the audio data using a Bluetooth protocol and transmits it to the earphone with low delay. When Wi-Fi is used, the audio transmission unit realizes high-speed data transfer and transmits the audio data to the earphone with high quality. This allows the user to listen to audio in other languages ​​in real time. Furthermore, the audio transmission unit has a technology for minimizing delays in audio data. For example, buffering and error checking of audio data is performed to prevent interruption and delay of audio data. In addition, the audio transmission unit can also transmit audio to multiple earphones at the same time. This allows multiple users to listen to audio in other languages ​​at the same time. This allows the audio transmission unit to realize high-quality and real-time audio transmission for the user and support comfortable communication.

[0034] The voice collection unit can analyze the characteristics of the user's voice quality and speaking style. For example, the voice collection unit analyzes the user's voice quality based on frequency characteristics and timbre. For example, the voice collection unit can analyze the user's voice quality based on frequency characteristics of low, medium, and high tones. The voice collection unit can also analyze the characteristics of the user's speaking style. For example, the voice collection unit can analyze the user's speaking speed, intonation, accent, and the like. This allows for more accurate voice collection by analyzing the characteristics of the user's voice quality and speaking style. A part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice collection unit can input the user's voice quality data to the generation AI and cause the generation AI to perform voice quality analysis.

[0035] The voice selection unit can select a voice close to the user's voice quality from the voice data storage. The voice selection unit, for example, selects a voice closest to the user's voice quality from the voice data storage. For example, the voice selection unit selects a low-pitched voice for a user with a low-pitched voice quality. The voice selection unit can also select a mid-pitched voice for a user with a mid-pitched voice quality. Furthermore, the voice selection unit can also select a high-pitched voice for a user with a high-pitched voice quality. By selecting a voice close to the user's voice quality, the converted voice can also sound like the user. A part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the user's voice quality data to the generation AI and cause the generation AI to select the optimal voice.

[0036] The speech conversion unit can convert speech into another language using a generation AI. The speech conversion unit converts speech into another language using, for example, a generation AI. The generation AI converts speech into another language using, for example, a text generation AI (e.g., LLM). The generation AI converts speech into another language while retaining the user's voice quality and speaking characteristics. For example, when a user says "Hello," the generation AI converts the speech into "Hello." The generation AI takes speech data as input and outputs speech data in another language. For example, the generation AI inputs the user's speech data as a prompt and generates speech data in another language. As a result, by using the generation AI, speech can be converted into another language with high accuracy.

[0037] The audio transmission unit can transmit the converted audio to the earphone. The audio transmission unit transmits the converted audio to the earphone using, for example, wireless communication technology such as Bluetooth or Wi-Fi. For example, the audio transmission unit transmits the converted audio to the earphone using Bluetooth. The audio transmission unit can also transmit the converted audio to the earphone using Wi-Fi. Furthermore, the audio transmission unit can also transmit the converted audio to the earphone using a wired connection. For example, the audio transmission unit transmits the converted audio to the earphone using an earphone jack. As a result, by transmitting the converted audio to the earphone, the user can listen to audio in another language in real time. A part or all of the above-mentioned processing in the audio transmission unit may be performed using, for example, AI, or may be performed without using AI. For example, the audio transmission unit can input the converted audio data to the generation AI and cause the generation AI to execute the optimal transmission method.

[0038] The voice collection unit can analyze the user's past voice data and select the optimal collection method. The voice collection unit, for example, analyzes the user's past voice data and selects the optimal collection method. For example, if the user has used a lot of high-pitched voice in the past, the voice collection unit selects a microphone setting specialized for the high-pitched range. In addition, if the user has used a lot of low-pitched voice in the past, the voice collection unit can also select a microphone setting specialized for the low-pitched range. Furthermore, if the user has collected voice in a specific environment in the past, the voice collection unit can also select a collection method optimal for that environment. In this way, the optimal voice collection method can be selected by analyzing the user's past voice data. Part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI or may be performed without using AI. For example, the voice collection unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal collection method.

[0039] The voice collection unit can collect clear voice by filtering the user's current environmental sound when collecting voice. For example, when the user is in a noisy environment, the voice collection unit uses noise canceling technology to filter the environmental sound. For example, the voice collection unit uses noise canceling technology to reduce ambient noise and collect the user's voice clearly. In addition, when the user is in a quiet environment, the voice collection unit can also collect clear voice with minimal environmental sound. For example, the voice collection unit uses a microphone specialized for voice collection in a quiet environment to collect clear voice. Furthermore, when the user is moving, the voice collection unit can also collect clear voice by filtering wind noise and traffic noise. For example, the voice collection unit uses a filtering algorithm to reduce environmental sound during movement to collect clear voice. As a result, clear voice can be collected by filtering the environmental sound. A part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the audio collection unit can input environmental sound data to the generation AI and have the generation AI perform a filtering process.

[0040] The voice collection unit can collect highly relevant voices preferentially in consideration of the geographical location information of the user when collecting voices. For example, when the user is in a specific area, the voice collection unit collects voices in consideration of the dialect or accent of the area. For example, the voice collection unit preferentially collects voices with a dialect or accent of the specific area. In addition, when the user is traveling, the voice collection unit can also preferentially collect voices related to the language or culture of the destination. For example, the voice collection unit collects voices related to the language or culture of the destination. In addition, when the user is at home, the voice collection unit can also preferentially collect voices related to daily conversations. For example, the voice collection unit collects voices related to daily conversations. As a result, highly relevant voices can be preferentially collected by considering the geographical location information. A part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice collection unit can input the geographical location information of the user to the generation AI and cause the generation AI to select highly relevant voices.

[0041] The voice collection unit can analyze the user's social media activity and collect related voices when collecting voices. For example, when the user is talking about a specific topic on social media, the voice collection unit collects voices related to the topic. For example, the voice collection unit collects voices related to topics that are trending on social media. In addition, when the user is participating in a specific event on social media, the voice collection unit can also collect voices related to the event. For example, the voice collection unit collects voices related to the event. Furthermore, when the user belongs to a specific group on social media, the voice collection unit can also collect voices related to the group. For example, the voice collection unit collects voices related to the group. In this way, it is possible to collect related voices by analyzing social media activities. Part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice collection unit can input the user's social media data to the generation AI and cause the generation AI to select related voices.

[0042] The voice selection unit can select an appropriate voice by referring to the user's past voice data when selecting a voice. The voice selection unit, for example, selects an appropriate voice by referring to the user's past voice data. For example, the voice selection unit selects an optimal voice based on voice data used by the user in the past. The voice selection unit can also select a voice having a specific tone or pitch from the user's past voice data. Furthermore, the voice selection unit can also analyze the user's past voice data and select the most natural voice. In this way, the optimal voice can be selected by referring to the past voice data. A part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI or may be performed without using AI. For example, the voice selection unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal voice.

[0043] The voice selection unit can select a voice by taking into consideration the fluctuation of the user's current voice quality when selecting a voice. For example, when the user's voice quality is fluctuating, the voice selection unit selects an optimal voice according to the fluctuation. For example, when the user's voice quality is tired, the voice selection unit selects a voice with a relaxed tone. In addition, when the user's voice quality is excited, the voice selection unit can also select a voice with an energetic tone. Furthermore, when the user's voice quality is fluctuating, the voice selection unit can also select an optimal voice according to the fluctuation. As a result, by taking into consideration the fluctuation of the voice quality, a more natural voice can be selected. Part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the user's voice quality data to the generation AI and cause the generation AI to select a voice based on the fluctuation of the voice quality.

[0044] The voice selection unit can preferentially select a highly relevant voice in consideration of the geographical location information of the user when selecting a voice. For example, when the user is in a specific region, the voice selection unit selects a voice in consideration of the dialect or accent of the region. For example, the voice selection unit preferentially selects a voice with a dialect or accent of a specific region. In addition, when the user is traveling, the voice selection unit can also preferentially select a voice related to the language or culture of the destination. For example, the voice selection unit selects a voice related to the language or culture of the destination. Furthermore, when the user is at home, the voice selection unit can also preferentially select a voice related to daily conversation. For example, the voice selection unit selects a voice related to daily conversation. As a result, by considering the geographical location information, it is possible to preferentially select a highly relevant voice. A part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the geographical location information of the user to the generation AI and cause the generation AI to select a highly relevant voice.

[0045] The voice selection unit can analyze the user's social media activity and select related voices when selecting voices. For example, when the user is talking about a specific topic on social media, the voice selection unit selects voices related to the topic. For example, the voice selection unit selects voices related to a topic that is a hot topic on social media. In addition, when the user is participating in a specific event on social media, the voice selection unit can also select voices related to the event. For example, the voice selection unit selects voices related to the event. Furthermore, when the user belongs to a specific group on social media, the voice selection unit can also select voices related to the group. For example, the voice selection unit selects voices related to the group. In this way, it is possible to select related voices by analyzing social media activities. Part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the user's social media data to the generation AI and cause the generation AI to select related voices.

[0046] The voice conversion unit can select an appropriate conversion method by referring to the user's past voice data at the time of voice conversion. The voice conversion unit, for example, selects an appropriate conversion method by referring to the user's past voice data. For example, the voice conversion unit selects the optimal conversion method based on the voice data used by the user in the past. The voice conversion unit can also select a voice having a specific tone or pitch from the user's past voice data. Furthermore, the voice conversion unit can also analyze the user's past voice data and select the most natural conversion method. In this way, the optimal voice conversion method can be selected by referring to the past voice data. A part or all of the above-mentioned processing in the voice conversion unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal conversion method.

[0047] The voice conversion unit can perform voice conversion while taking into account the fluctuation of the user's current voice quality. For example, when the user's voice quality fluctuates, the voice conversion unit performs optimal conversion according to the fluctuation. For example, when the user's voice quality is tired, the voice conversion unit converts the user's voice into another language with a relaxed tone. In addition, when the user's voice quality is excited, the voice conversion unit can also convert the user's voice into another language with an energetic tone. Furthermore, when the user's voice quality is fluctuating, the voice conversion unit can also perform optimal conversion according to the fluctuation. As a result, more natural voice conversion is possible by taking into account the fluctuation of the voice quality. Part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the user's voice quality data into the generation AI and cause the generation AI to perform conversion based on the fluctuation of the voice quality.

[0048] The voice conversion unit can convert a highly relevant voice preferentially in consideration of the geographical location information of the user during voice conversion. For example, when the user is in a specific region, the voice conversion unit converts the voice in consideration of the dialect or accent of the region. For example, the voice conversion unit converts a voice having a dialect or accent of a specific region preferentially. In addition, when the user is traveling, the voice conversion unit can also convert a voice related to the language or culture of the destination preferentially. For example, the voice conversion unit converts a voice related to the language or culture of the destination. Furthermore, when the user is at home, the voice conversion unit can also convert a voice related to daily conversation preferentially. For example, the voice conversion unit converts a voice related to daily conversation. As a result, by considering the geographical location information, it is possible to convert a highly relevant voice preferentially. A part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the geographical location information of the user to the generation AI and cause the generation AI to select a highly relevant voice.

[0049] The voice conversion unit can analyze the social media activity of the user and convert the related voice during the voice conversion. For example, when the user is talking about a specific topic on social media, the voice conversion unit converts the voice related to the topic. For example, the voice conversion unit converts the voice related to the topic that is a hot topic on social media. In addition, when the user is participating in a specific event on social media, the voice conversion unit can also convert the voice related to the event. For example, the voice conversion unit converts the voice related to the event. Furthermore, when the user belongs to a specific group on social media, the voice conversion unit can also convert the voice related to the group. For example, the voice conversion unit converts the voice related to the group. In this way, the related voice can be converted by analyzing the social media activity. Part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the social media data of the user to the generation AI and cause the generation AI to select the related voice.

[0050] The voice transmission unit can select an appropriate transmission method by referring to the user's past voice data when transmitting voice. The voice transmission unit, for example, selects an appropriate transmission method by referring to the user's past voice data. For example, the voice transmission unit selects the optimal transmission method based on the voice data used by the user in the past. The voice transmission unit can also select a voice having a specific tone or pitch from the user's past voice data. Furthermore, the voice transmission unit can analyze the user's past voice data and select the most natural transmission method. In this way, the optimal voice transmission method can be selected by referring to the past voice data. A part or all of the above-mentioned processing in the voice transmission unit may be performed, for example, using AI or may be performed without using AI. For example, the voice transmission unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal transmission method.

[0051] The voice transmission unit can transmit clear voice by filtering the user's current environmental sound when transmitting voice. For example, when the user is in a noisy environment, the voice transmission unit filters the environmental sound using noise canceling technology. For example, the voice transmission unit uses noise canceling technology to reduce ambient noise and transmit the user's voice clearly. In addition, when the user is in a quiet environment, the voice transmission unit can also transmit clear voice with minimal environmental sound. For example, the voice transmission unit transmits clear voice using a microphone specialized for voice transmission in a quiet environment. Furthermore, when the user is moving, the voice transmission unit can also transmit clear voice by filtering wind noise and traffic noise. For example, the voice transmission unit transmits clear voice using a filtering algorithm for reducing environmental sound during movement. As a result, clear voice can be transmitted by filtering the environmental sound. A part or all of the above-mentioned processing in the voice transmission unit may be performed using, for example, AI, or may be performed without using AI. For example, the audio transmission unit can input environmental sound data to the generation AI and have the generation AI perform a filtering process.

[0052] The voice transmission unit can preferentially transmit highly relevant voices in consideration of the geographical location information of the user when transmitting voices. For example, when the user is in a specific region, the voice transmission unit transmits voices in consideration of the dialect or accent of the region. For example, the voice transmission unit preferentially transmits voices with a dialect or accent of a specific region. In addition, when the user is traveling, the voice transmission unit can also preferentially transmit voices related to the language or culture of the destination. For example, the voice transmission unit transmits voices related to the language or culture of the destination. In addition, when the user is at home, the voice transmission unit can also preferentially transmit voices related to daily conversations. For example, the voice transmission unit transmits voices related to daily conversations. As a result, highly relevant voices can be preferentially transmitted by considering the geographical location information. A part or all of the above-mentioned processing in the voice transmission unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice transmission unit can input the geographical location information of the user to the generation AI and cause the generation AI to select highly relevant voices.

[0053] The voice transmission unit can analyze the social media activity of the user at the time of voice transmission and transmit the related voice. For example, when the user is talking about a specific topic on social media, the voice transmission unit transmits the voice related to the topic. For example, the voice transmission unit transmits the voice related to the topic that is a hot topic on social media. In addition, when the user is participating in a specific event on social media, the voice transmission unit can also transmit the voice related to the event. For example, the voice transmission unit transmits the voice related to the event. Furthermore, when the user belongs to a specific group on social media, the voice transmission unit can also transmit the voice related to the group. For example, the voice transmission unit transmits the voice related to the group. In this way, the related voice can be transmitted by analyzing the social media activity. A part or all of the above-mentioned processing in the voice transmission unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice transmission unit can input the social media data of the user to the generation AI and cause the generation AI to select the related voice.

[0054] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0055] The voice conversion system may further include a schedule management unit that manages schedule information of the user. The schedule management unit obtains information from the user's calendar or planner, and adjusts the timing and content of the voice conversion. For example, if the user is attending an important meeting, the system may preferentially convert technical terms and phrases related to the content of the meeting. Also, if the user is on vacation, the system may convert the voice in a relaxed tone. Furthermore, if the user is traveling, the system may preferentially convert voice related to the language and culture of the destination. This enables appropriate voice conversion according to the user's schedule.

[0056] The voice conversion system may further include a hobby analysis unit that analyzes the user's hobbies and interests. The hobby analysis unit identifies the user's hobbies and interests from past voice data and social media activities, and customizes the content of the voice conversion. For example, if the user is interested in music, the system may preferentially convert terms and phrases related to music. Also, if the user is interested in sports, the system may preferentially convert topics related to sports. Furthermore, if the user likes traveling, the system may preferentially convert information related to travel. This enables personalized voice conversion according to the user's hobbies and interests.

[0057] The speech conversion system may further include a learning tracker that tracks the user's learning progress. The learning tracker monitors the progress of the language or skill that the user is learning and adjusts the content of the speech conversion. For example, if the user is learning a new language, the system may preferentially convert basic phrases and words related to that language. Also, if the user is acquiring a specific skill, the system may preferentially convert technical terms related to that skill. Furthermore, according to the user's learning progress, the system may adjust the difficulty level and provide more advanced content. This enables effective speech conversion according to the user's learning progress.

[0058] The voice conversion system may further include a conversation history analysis unit that analyzes the user's past conversation history. The conversation history analysis unit analyzes the user's past conversation content and improves the accuracy of voice conversion. For example, if the user has frequently used a specific phrase in the past, the system may preferentially convert that phrase. Also, if the user has spoken a lot about a specific topic in the past, the system may preferentially convert terms related to that topic. Furthermore, based on the user's past conversation history, the system may learn the user's speaking patterns and provide more natural voice conversion. This enables highly accurate voice conversion based on the user's past conversation history.

[0059] The speech conversion system may further include a feedback collection unit that collects real-time feedback from the user. The feedback collection unit allows the user to provide real-time feedback on the results of speech conversion. For example, if the user is satisfied with the converted speech, the system may optimize the conversion algorithm based on the feedback. Also, if the user is dissatisfied with the conversion result, the system may identify improvements based on the feedback and incorporate them into the next conversion. Furthermore, if the user provides specific feedback on the conversion of a particular phrase or term, the system may improve the conversion accuracy based on the feedback. This allows flexible speech conversion based on real-time user feedback.

[0060] The speech conversion system can further customize the contents of the speech conversion taking into account the geographic location information of the user. For example, if the user is in a particular area, phrases and terms related to the culture and customs of that area can be preferentially converted. Also, if the user is traveling, information related to tourist attractions and famous places of the visited area can be preferentially converted. Furthermore, if the user is on a business trip, speech related to the business customs and technical terms of the visited area can be preferentially converted. This enables appropriate speech conversion according to the geographic location information of the user.

[0061] The process flow of the first embodiment will be briefly described below.

[0062] Step 1: The voice collection unit collects the user's voice. The user's voice includes speaking characteristics and voice quality. The voice collection unit collects the user's voice using a microphone, and can also analyze the user's voice quality and speaking characteristics. For example, the voice collection unit analyzes the user's voice quality based on frequency characteristics and timbre. Step 2: The voice selection unit selects a voice from the voice data storage that is closest to the user's voice quality based on the voice data collected by the voice collection unit. Voices with various voice qualities are stored in the voice data storage, and the voice selection unit selects the voice that is closest to the user's voice quality. For example, for a user with a low-pitched voice quality, a low-pitched voice is selected. Step 3: The speech conversion unit uses a generation AI to convert the speech selected by the speech selection unit into another language. The generation AI uses a text generation AI (e.g., LLM) to convert the speech into another language, preserving the user's voice quality and speaking characteristics. For example, if the user says "Hello," the speech is converted into "Hello." Step 4: The audio transmission unit transmits the audio converted by the audio conversion unit to the earphones. The audio transmission unit transmits the converted audio to the earphones using wireless communication technology such as Bluetooth or Wi-Fi. This allows the user to listen to audio in other languages ​​in real time.

[0063] (Example 2) The voice conversion system according to the embodiment of the present invention is a system that converts the user's voice into another language in real time and transmits it through earphones. This voice conversion system collects the user's voice, selects a voice similar to the user's voice quality from a voice database, converts the voice into another language using a generation AI, and transmits the converted voice to the earphones. This allows the user to overcome language barriers and freely enjoy communication while maintaining their individuality. For example, when a user says "Hello," the voice data is collected and a voice similar to the user's voice quality is selected from the voice database. Next, the generation AI converts the selected voice into "Hello," and the converted voice is transmitted to the earphones. This allows the user to speak in Japanese, but the other person will hear it in English. This allows smooth communication between different cultures and promotes international exchange. This allows the voice conversion system to convert the user's voice into another language in real time and freely enjoy communication while maintaining their individuality.

[0064] The voice conversion system according to the embodiment includes a voice collection unit, a voice selection unit, a voice conversion unit, and a voice transmission unit. The voice collection unit collects the user's voice. The user's voice includes, for example, speech characteristics and voice quality, but is not limited to such examples. The voice collection unit collects the user's voice using, for example, a microphone. The voice collection unit can also analyze the user's voice quality and speech characteristics. For example, the voice collection unit analyzes the user's voice quality based on frequency characteristics and timbre. The voice selection unit selects a voice close to the user's voice quality from the voice data storage based on the voice data collected by the voice collection unit. Voices of various voice qualities are stored in the voice data storage, and the voice selection unit selects a voice closest to the user's voice quality. For example, the voice selection unit selects a low-pitched voice for a user with a low-pitched voice quality. The voice conversion unit converts the voice selected by the voice selection unit into another language using a generation AI. The generation AI converts the voice into another language using, for example, a text generation AI (for example, LLM). The generation AI converts the voice into another language while preserving the user's voice quality and speaking characteristics. For example, when the user says "Hello," the generation AI converts the voice into "Hello." The voice transmission unit transmits the voice converted by the voice conversion unit to the earphones. The voice transmission unit transmits the converted voice to the earphones using wireless communication technology such as Bluetooth or Wi-Fi. This allows the user to listen to the voice in another language in real time. This allows the voice conversion system according to the embodiment to convert the user's voice into another language in real time, allowing the user to freely enjoy communication while preserving their individuality.

[0065] The voice collecting unit collects the user's voice. The user's voice includes, for example, speech characteristics and voice quality, but is not limited to such examples. The voice collecting unit collects the user's voice using, for example, a microphone. Specifically, a high-sensitivity microphone can be used to clearly collect the user's voice. The microphone has a noise canceling function, which removes ambient noise and accurately collects only the user's voice. The voice collecting unit can also analyze the user's voice quality and speech characteristics. For example, the voice collecting unit analyzes the user's voice quality based on frequency characteristics and timbre. Specifically, the voice signal is digitized and the frequency spectrum is analyzed to extract the characteristics of the user's voice. This allows the voice collecting unit to grasp the characteristics of the user's voice, such as the pitch, timbre, and rhythm, in detail. Furthermore, the voice collecting unit can also analyze the user's speech pattern and accent. For example, it is possible to analyze how the user pronounces a specific word and what kind of rhythm he or she speaks with, and improve the accuracy of voice conversion based on this information. This allows the voice collecting unit to collect and analyze the user's voice with high accuracy.

[0066] The voice selection unit selects a voice close to the user's voice quality from the voice data storage based on the voice data collected by the voice collection unit. Voices of various voice qualities are stored in the voice data storage, and the voice selection unit selects a voice closest to the user's voice quality. Specifically, the voice selection unit analyzes the voice data stored in the voice data storage and searches for a voice having characteristics matching the user's voice quality. For example, the voice selection unit analyzes the frequency characteristics and timbre of the voice data stored in the voice data storage and identifies a voice matching the user's voice quality. Furthermore, the voice selection unit selects a voice taking into consideration the characteristics of the user's speaking style. For example, if the user has a low-pitched voice quality, the voice selection unit selects a low-pitched voice. Also, if the user speaks with a specific accent or rhythm, the voice selection unit selects a voice having these characteristics. In this way, the voice selection unit can select a voice closest to the characteristics of the user's voice quality and speaking style, and improve the accuracy of voice conversion.

[0067] The speech conversion unit converts the speech selected by the speech selection unit into another language using the generation AI. The generation AI converts the speech into another language using, for example, a text generation AI (for example, LLM). Specifically, the generation AI converts the speech data into text data, and generates a speech in another language based on the text data. The generation AI converts the speech into another language while preserving the user's voice quality and speaking characteristics. For example, when the user says "Hello," the generation AI converts the speech into "Hello." The generation AI analyzes the speech data and extracts the user's voice quality and speaking characteristics. Next, the generation AI generates a speech in another language based on the extracted characteristics. This allows the generation AI to generate a speech in another language while preserving the user's individuality. Furthermore, the generation AI can continuously learn to improve the accuracy of the speech conversion. For example, the generation AI updates the speech conversion model based on the user's feedback, and improves the conversion accuracy. This allows the speech conversion unit to convert the speech into another language with high accuracy while preserving the user's voice quality and speaking characteristics.

[0068] The audio transmission unit transmits the audio converted by the audio conversion unit to the earphone. The audio transmission unit transmits the converted audio to the earphone using a wireless communication technology such as Bluetooth or Wi-Fi. Specifically, the audio transmission unit transmits the converted audio data to the earphone via the wireless communication module. When Bluetooth is used, the audio transmission unit compresses the audio data using a Bluetooth protocol and transmits it to the earphone with low delay. When Wi-Fi is used, the audio transmission unit realizes high-speed data transfer and transmits the audio data to the earphone with high quality. This allows the user to listen to audio in other languages ​​in real time. Furthermore, the audio transmission unit has a technology for minimizing delays in audio data. For example, buffering and error checking of audio data is performed to prevent interruption and delay of audio data. In addition, the audio transmission unit can also transmit audio to multiple earphones at the same time. This allows multiple users to listen to audio in other languages ​​at the same time. This allows the audio transmission unit to realize high-quality and real-time audio transmission for the user and support comfortable communication.

[0069] The voice collection unit can analyze the characteristics of the user's voice quality and speaking style. For example, the voice collection unit analyzes the user's voice quality based on frequency characteristics and timbre. For example, the voice collection unit can analyze the user's voice quality based on frequency characteristics of low, medium, and high tones. The voice collection unit can also analyze the characteristics of the user's speaking style. For example, the voice collection unit can analyze the user's speaking speed, intonation, accent, and the like. This allows for more accurate voice collection by analyzing the characteristics of the user's voice quality and speaking style. A part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice collection unit can input the user's voice quality data to the generation AI and cause the generation AI to perform voice quality analysis.

[0070] The voice selection unit can select a voice close to the user's voice quality from the voice data storage. The voice selection unit, for example, selects a voice closest to the user's voice quality from the voice data storage. For example, the voice selection unit selects a low-pitched voice for a user with a low-pitched voice quality. The voice selection unit can also select a mid-pitched voice for a user with a mid-pitched voice quality. Furthermore, the voice selection unit can also select a high-pitched voice for a user with a high-pitched voice quality. By selecting a voice close to the user's voice quality, the converted voice can also sound like the user. A part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the user's voice quality data to the generation AI and cause the generation AI to select the optimal voice.

[0071] The speech conversion unit can convert speech into another language using a generation AI. The speech conversion unit converts speech into another language using, for example, a generation AI. The generation AI converts speech into another language using, for example, a text generation AI (e.g., LLM). The generation AI converts speech into another language while retaining the user's voice quality and speaking characteristics. For example, when a user says "Hello," the generation AI converts the speech into "Hello." The generation AI takes speech data as input and outputs speech data in another language. For example, the generation AI inputs the user's speech data as a prompt and generates speech data in another language. As a result, by using the generation AI, speech can be converted into another language with high accuracy.

[0072] The audio transmission unit can transmit the converted audio to the earphone. The audio transmission unit transmits the converted audio to the earphone using, for example, wireless communication technology such as Bluetooth or Wi-Fi. For example, the audio transmission unit transmits the converted audio to the earphone using Bluetooth. The audio transmission unit can also transmit the converted audio to the earphone using Wi-Fi. Furthermore, the audio transmission unit can also transmit the converted audio to the earphone using a wired connection. For example, the audio transmission unit transmits the converted audio to the earphone using an earphone jack. As a result, by transmitting the converted audio to the earphone, the user can listen to audio in another language in real time. A part or all of the above-mentioned processing in the audio transmission unit may be performed using, for example, AI, or may be performed without using AI. For example, the audio transmission unit can input the converted audio data to the generation AI and cause the generation AI to execute the optimal transmission method.

[0073] The voice collection unit can estimate the user's emotion and adjust the timing of voice collection based on the estimated user's emotion. The voice collection unit, for example, estimates the user's emotion and adjusts the timing of voice collection based on the estimated user's emotion. For example, when the user is relaxed, the voice collection unit reduces the frequency of voice collection to maintain a natural conversation flow. In addition, when the user is nervous, the voice collection unit can also collect voice at short intervals and provide feedback in real time. Furthermore, when the user is excited, the voice collection unit can collect voice at the peak of the emotion and perform more accurate emotion analysis. This enables more natural voice collection by adjusting the timing of voice collection according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice collection unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice collection unit can input a user's emotion data to the generation AI and have the generation AI perform emotion estimation.

[0074] The voice collection unit can analyze the user's past voice data and select the optimal collection method. The voice collection unit, for example, analyzes the user's past voice data and selects the optimal collection method. For example, if the user has used a lot of high-pitched voice in the past, the voice collection unit selects a microphone setting specialized for the high-pitched range. In addition, if the user has used a lot of low-pitched voice in the past, the voice collection unit can also select a microphone setting specialized for the low-pitched range. Furthermore, if the user has collected voice in a specific environment in the past, the voice collection unit can also select a collection method optimal for that environment. In this way, the optimal voice collection method can be selected by analyzing the user's past voice data. Part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI or may be performed without using AI. For example, the voice collection unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal collection method.

[0075] The voice collection unit can collect clear voice by filtering the user's current environmental sound when collecting voice. For example, when the user is in a noisy environment, the voice collection unit uses noise canceling technology to filter the environmental sound. For example, the voice collection unit uses noise canceling technology to reduce ambient noise and collect the user's voice clearly. In addition, when the user is in a quiet environment, the voice collection unit can also collect clear voice with minimal environmental sound. For example, the voice collection unit uses a microphone specialized for voice collection in a quiet environment to collect clear voice. Furthermore, when the user is moving, the voice collection unit can also collect clear voice by filtering wind noise and traffic noise. For example, the voice collection unit uses a filtering algorithm to reduce environmental sound during movement to collect clear voice. As a result, clear voice can be collected by filtering the environmental sound. A part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the audio collection unit can input environmental sound data to the generation AI and have the generation AI perform a filtering process.

[0076] The voice collection unit can estimate the user's emotion and determine the priority of the voice to be collected based on the estimated user's emotion. The voice collection unit, for example, estimates the user's emotion and determines the priority of the voice to be collected based on the estimated user's emotion. For example, when the user is excited, the voice collection unit preferentially collects the voice at the peak of the emotion. In addition, when the user is relaxed, the voice collection unit can also reduce the frequency of voice collection to maintain a natural conversation flow. Furthermore, when the user is nervous, the voice collection unit can also collect voice at short intervals and provide feedback in real time. In this way, by determining the priority of the voice according to the user's emotion, it is possible to preferentially collect more important voice. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice collection unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice collection unit can input a user's emotion data to the generation AI and have the generation AI perform emotion estimation.

[0077] The voice collection unit can collect highly relevant voices preferentially in consideration of the geographical location information of the user when collecting voices. For example, when the user is in a specific area, the voice collection unit collects voices in consideration of the dialect or accent of the area. For example, the voice collection unit preferentially collects voices with a dialect or accent of the specific area. In addition, when the user is traveling, the voice collection unit can also preferentially collect voices related to the language or culture of the destination. For example, the voice collection unit collects voices related to the language or culture of the destination. In addition, when the user is at home, the voice collection unit can also preferentially collect voices related to daily conversations. For example, the voice collection unit collects voices related to daily conversations. As a result, highly relevant voices can be preferentially collected by considering the geographical location information. A part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice collection unit can input the geographical location information of the user to the generation AI and cause the generation AI to select highly relevant voices.

[0078] The voice collection unit can analyze the user's social media activity and collect related voices when collecting voices. For example, when the user is talking about a specific topic on social media, the voice collection unit collects voices related to the topic. For example, the voice collection unit collects voices related to topics that are trending on social media. In addition, when the user is participating in a specific event on social media, the voice collection unit can also collect voices related to the event. For example, the voice collection unit collects voices related to the event. Furthermore, when the user belongs to a specific group on social media, the voice collection unit can also collect voices related to the group. For example, the voice collection unit collects voices related to the group. In this way, it is possible to collect related voices by analyzing social media activities. Part or all of the above-mentioned processing in the voice collection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice collection unit can input the user's social media data to the generation AI and cause the generation AI to select related voices.

[0079] The voice selection unit can estimate the user's emotion and adjust the criteria for the voice to be selected based on the estimated user's emotion. The voice selection unit, for example, estimates the user's emotion and adjusts the criteria for the voice to be selected based on the estimated user's emotion. For example, when the user is relaxed, the voice selection unit preferentially selects a voice with a soft tone. In addition, when the user is nervous, the voice selection unit can also preferentially select a voice with a calm tone. Furthermore, when the user is excited, the voice selection unit can also preferentially select a voice with an energetic tone. In this way, by adjusting the criteria for the voice according to the user's emotion, a more appropriate voice can be selected. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice selection unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice selection unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0080] The voice selection unit can select an appropriate voice by referring to the user's past voice data when selecting a voice. The voice selection unit, for example, selects an appropriate voice by referring to the user's past voice data. For example, the voice selection unit selects an optimal voice based on voice data used by the user in the past. The voice selection unit can also select a voice having a specific tone or pitch from the user's past voice data. Furthermore, the voice selection unit can also analyze the user's past voice data and select the most natural voice. In this way, the optimal voice can be selected by referring to the past voice data. A part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI or may be performed without using AI. For example, the voice selection unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal voice.

[0081] The voice selection unit can select a voice by taking into consideration the fluctuation of the user's current voice quality when selecting a voice. For example, when the user's voice quality is fluctuating, the voice selection unit selects an optimal voice according to the fluctuation. For example, when the user's voice quality is tired, the voice selection unit selects a voice with a relaxed tone. In addition, when the user's voice quality is excited, the voice selection unit can also select a voice with an energetic tone. Furthermore, when the user's voice quality is fluctuating, the voice selection unit can also select an optimal voice according to the fluctuation. As a result, by taking into consideration the fluctuation of the voice quality, a more natural voice can be selected. Part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the user's voice quality data to the generation AI and cause the generation AI to select a voice based on the fluctuation of the voice quality.

[0082] The voice selection unit can estimate the user's emotion and determine the priority of the voice to be selected based on the estimated user's emotion. The voice selection unit, for example, estimates the user's emotion and determines the priority of the voice to be selected based on the estimated user's emotion. For example, when the user is relaxed, the voice selection unit preferentially selects a voice with a soft tone. In addition, when the user is nervous, the voice selection unit can also preferentially select a voice with a calm tone. Furthermore, when the user is excited, the voice selection unit can also preferentially select a voice with an energetic tone. In this way, by determining the priority of the voice according to the user's emotion, a more appropriate voice can be selected. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice selection unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice selection unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0083] The voice selection unit can preferentially select a highly relevant voice in consideration of the geographical location information of the user when selecting a voice. For example, when the user is in a specific region, the voice selection unit selects a voice in consideration of the dialect or accent of the region. For example, the voice selection unit preferentially selects a voice with a dialect or accent of a specific region. In addition, when the user is traveling, the voice selection unit can also preferentially select a voice related to the language or culture of the destination. For example, the voice selection unit selects a voice related to the language or culture of the destination. Furthermore, when the user is at home, the voice selection unit can also preferentially select a voice related to daily conversation. For example, the voice selection unit selects a voice related to daily conversation. As a result, by considering the geographical location information, it is possible to preferentially select a highly relevant voice. A part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the geographical location information of the user to the generation AI and cause the generation AI to select a highly relevant voice.

[0084] The voice selection unit can analyze the user's social media activity and select related voices when selecting voices. For example, when the user is talking about a specific topic on social media, the voice selection unit selects voices related to the topic. For example, the voice selection unit selects voices related to a topic that is a hot topic on social media. In addition, when the user is participating in a specific event on social media, the voice selection unit can also select voices related to the event. For example, the voice selection unit selects voices related to the event. Furthermore, when the user belongs to a specific group on social media, the voice selection unit can also select voices related to the group. For example, the voice selection unit selects voices related to the group. In this way, it is possible to select related voices by analyzing social media activities. Part or all of the above-mentioned processing in the voice selection unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice selection unit can input the user's social media data to the generation AI and cause the generation AI to select related voices.

[0085] The voice conversion unit can estimate the user's emotion and adjust the expression method of the voice to be converted based on the estimated user's emotion. The voice conversion unit, for example, estimates the user's emotion and adjusts the expression method of the voice to be converted based on the estimated user's emotion. For example, when the user is relaxed, the voice conversion unit converts into another language in a soft tone. Also, when the user is nervous, the voice conversion unit can convert into another language in a calm tone. Furthermore, when the user is excited, the voice conversion unit can convert into another language in an energetic tone. This allows for more natural voice conversion by adjusting the expression method of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice conversion unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0086] The voice conversion unit can select an appropriate conversion method by referring to the user's past voice data at the time of voice conversion. The voice conversion unit, for example, selects an appropriate conversion method by referring to the user's past voice data. For example, the voice conversion unit selects the optimal conversion method based on the voice data used by the user in the past. The voice conversion unit can also select a voice having a specific tone or pitch from the user's past voice data. Furthermore, the voice conversion unit can also analyze the user's past voice data and select the most natural conversion method. In this way, the optimal voice conversion method can be selected by referring to the past voice data. A part or all of the above-mentioned processing in the voice conversion unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal conversion method.

[0087] The voice conversion unit can perform voice conversion while taking into account the fluctuation of the user's current voice quality. For example, when the user's voice quality fluctuates, the voice conversion unit performs optimal conversion according to the fluctuation. For example, when the user's voice quality is tired, the voice conversion unit converts the user's voice into another language with a relaxed tone. In addition, when the user's voice quality is excited, the voice conversion unit can also convert the user's voice into another language with an energetic tone. Furthermore, when the user's voice quality is fluctuating, the voice conversion unit can also perform optimal conversion according to the fluctuation. As a result, more natural voice conversion is possible by taking into account the fluctuation of the voice quality. Part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the user's voice quality data into the generation AI and cause the generation AI to perform conversion based on the fluctuation of the voice quality.

[0088] The voice conversion unit can estimate the user's emotion and determine the priority of the voice to be converted based on the estimated user's emotion. The voice conversion unit, for example, estimates the user's emotion and determines the priority of the voice to be converted based on the estimated user's emotion. For example, when the user is relaxed, the voice conversion unit preferentially converts a voice with a soft tone. In addition, when the user is nervous, the voice conversion unit can also preferentially convert a voice with a calm tone. Furthermore, when the user is excited, the voice conversion unit can also preferentially convert a voice with an energetic tone. In this way, by determining the priority of the voice according to the user's emotion, it is possible to convert a more appropriate voice. The estimation of the emotion is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, the generation AI, or may be performed without using the generation AI. For example, the voice conversion unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0089] The voice conversion unit can convert a highly relevant voice preferentially in consideration of the geographical location information of the user during voice conversion. For example, when the user is in a specific region, the voice conversion unit converts the voice in consideration of the dialect or accent of the region. For example, the voice conversion unit converts a voice having a dialect or accent of a specific region preferentially. In addition, when the user is traveling, the voice conversion unit can also convert a voice related to the language or culture of the destination preferentially. For example, the voice conversion unit converts a voice related to the language or culture of the destination. Furthermore, when the user is at home, the voice conversion unit can also convert a voice related to daily conversation preferentially. For example, the voice conversion unit converts a voice related to daily conversation. As a result, by considering the geographical location information, it is possible to convert a highly relevant voice preferentially. A part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the geographical location information of the user to the generation AI and cause the generation AI to select a highly relevant voice.

[0090] The voice conversion unit can analyze the social media activity of the user and convert the related voice during the voice conversion. For example, when the user is talking about a specific topic on social media, the voice conversion unit converts the voice related to the topic. For example, the voice conversion unit converts the voice related to the topic that is a hot topic on social media. In addition, when the user is participating in a specific event on social media, the voice conversion unit can also convert the voice related to the event. For example, the voice conversion unit converts the voice related to the event. Furthermore, when the user belongs to a specific group on social media, the voice conversion unit can also convert the voice related to the group. For example, the voice conversion unit converts the voice related to the group. In this way, the related voice can be converted by analyzing the social media activity. Part or all of the above-mentioned processing in the voice conversion unit may be performed using, for example, a generation AI, or may be performed without using a generation AI. For example, the voice conversion unit can input the social media data of the user to the generation AI and cause the generation AI to select the related voice.

[0091] The voice transmission unit can estimate the user's emotion and adjust the method of voice transmission based on the estimated user's emotion. The voice transmission unit, for example, estimates the user's emotion and adjusts the method of voice transmission based on the estimated user's emotion. For example, the voice transmission unit transmits voice in a soft tone when the user is relaxed. In addition, the voice transmission unit can also transmit voice in a calm tone when the user is nervous. Furthermore, the voice transmission unit can also transmit voice in an energetic tone when the user is excited. This enables more natural voice transmission by adjusting the method of voice transmission according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice transmission unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice transmission unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0092] The voice transmission unit can select an appropriate transmission method by referring to the user's past voice data when transmitting voice. The voice transmission unit, for example, selects an appropriate transmission method by referring to the user's past voice data. For example, the voice transmission unit selects the optimal transmission method based on the voice data used by the user in the past. The voice transmission unit can also select a voice having a specific tone or pitch from the user's past voice data. Furthermore, the voice transmission unit can analyze the user's past voice data and select the most natural transmission method. In this way, the optimal voice transmission method can be selected by referring to the past voice data. A part or all of the above-mentioned processing in the voice transmission unit may be performed, for example, using AI or may be performed without using AI. For example, the voice transmission unit can input the user's past voice data to the generation AI and cause the generation AI to select the optimal transmission method.

[0093] The voice transmission unit can transmit clear voice by filtering the user's current environmental sound when transmitting voice. For example, when the user is in a noisy environment, the voice transmission unit filters the environmental sound using noise canceling technology. For example, the voice transmission unit uses noise canceling technology to reduce ambient noise and transmit the user's voice clearly. In addition, when the user is in a quiet environment, the voice transmission unit can also transmit clear voice with minimal environmental sound. For example, the voice transmission unit transmits clear voice using a microphone specialized for voice transmission in a quiet environment. Furthermore, when the user is moving, the voice transmission unit can also transmit clear voice by filtering wind noise and traffic noise. For example, the voice transmission unit transmits clear voice using a filtering algorithm for reducing environmental sound during movement. As a result, clear voice can be transmitted by filtering the environmental sound. A part or all of the above-mentioned processing in the voice transmission unit may be performed using, for example, AI, or may be performed without using AI. For example, the audio transmission unit can input environmental sound data to the generation AI and have the generation AI perform a filtering process.

[0094] The voice transmission unit can estimate the user's emotion and determine the priority of voice transmission based on the estimated user's emotion. The voice transmission unit, for example, estimates the user's emotion and determines the priority of voice transmission based on the estimated user's emotion. For example, when the user is relaxed, the voice transmission unit preferentially transmits a voice with a soft tone. In addition, when the user is nervous, the voice transmission unit can also preferentially transmit a voice with a calm tone. Furthermore, when the user is excited, the voice transmission unit can also preferentially transmit a voice with an energetic tone. In this way, by determining the priority of voice transmission according to the user's emotion, it is possible to transmit a more appropriate voice. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice transmission unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice transmission unit can input the user's emotion data to the generation AI and cause the generation AI to execute emotion estimation.

[0095] The voice transmission unit can preferentially transmit highly relevant voices in consideration of the geographical location information of the user when transmitting voices. For example, when the user is in a specific region, the voice transmission unit transmits voices in consideration of the dialect or accent of the region. For example, the voice transmission unit preferentially transmits voices with a dialect or accent of a specific region. In addition, when the user is traveling, the voice transmission unit can also preferentially transmit voices related to the language or culture of the destination. For example, the voice transmission unit transmits voices related to the language or culture of the destination. In addition, when the user is at home, the voice transmission unit can also preferentially transmit voices related to daily conversations. For example, the voice transmission unit transmits voices related to daily conversations. As a result, highly relevant voices can be preferentially transmitted by considering the geographical location information. A part or all of the above-mentioned processing in the voice transmission unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice transmission unit can input the geographical location information of the user to the generation AI and cause the generation AI to select highly relevant voices.

[0096] The voice transmission unit can analyze the social media activity of the user at the time of voice transmission and transmit the related voice. For example, when the user is talking about a specific topic on social media, the voice transmission unit transmits the voice related to the topic. For example, the voice transmission unit transmits the voice related to the topic that is a hot topic on social media. In addition, when the user is participating in a specific event on social media, the voice transmission unit can also transmit the voice related to the event. For example, the voice transmission unit transmits the voice related to the event. Furthermore, when the user belongs to a specific group on social media, the voice transmission unit can also transmit the voice related to the group. For example, the voice transmission unit transmits the voice related to the group. In this way, the related voice can be transmitted by analyzing the social media activity. A part or all of the above-mentioned processing in the voice transmission unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice transmission unit can input the social media data of the user to the generation AI and cause the generation AI to select the related voice.

[0097] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0098] The voice conversion system may further include a health monitoring unit that monitors the user's health condition. The health monitoring unit collects biometric information such as the user's heart rate, blood pressure, and body temperature, and evaluates the user's health condition based on this data. For example, if the user's heart rate is high, the system may determine that the user is nervous and adjust the tone of the voice conversion to a calmer tone. Also, if the user's body temperature is high, the system may consider that the user may be in poor health and reduce the frequency of voice collection. Furthermore, if the user's blood pressure is stable, the system may maintain the normal voice conversion process. This allows for flexible voice conversion according to the user's health condition.

[0099] The voice conversion system may further include a schedule management unit that manages schedule information of the user. The schedule management unit obtains information from the user's calendar or planner, and adjusts the timing and content of the voice conversion. For example, if the user is attending an important meeting, the system may preferentially convert technical terms and phrases related to the content of the meeting. Also, if the user is on vacation, the system may convert the voice in a relaxed tone. Furthermore, if the user is traveling, the system may preferentially convert voice related to the language and culture of the destination. This enables appropriate voice conversion according to the user's schedule.

[0100] The voice conversion system may further include a hobby analysis unit that analyzes the user's hobbies and interests. The hobby analysis unit identifies the user's hobbies and interests from past voice data and social media activities, and customizes the content of the voice conversion. For example, if the user is interested in music, the system may preferentially convert terms and phrases related to music. Also, if the user is interested in sports, the system may preferentially convert topics related to sports. Furthermore, if the user likes traveling, the system may preferentially convert information related to travel. This enables personalized voice conversion according to the user's hobbies and interests.

[0101] The speech conversion system may further include a learning tracker that tracks the user's learning progress. The learning tracker monitors the progress of the language or skill that the user is learning and adjusts the content of the speech conversion. For example, if the user is learning a new language, the system may preferentially convert basic phrases and words related to that language. Also, if the user is acquiring a specific skill, the system may preferentially convert technical terms related to that skill. Furthermore, according to the user's learning progress, the system may adjust the difficulty level and provide more advanced content. This enables effective speech conversion according to the user's learning progress.

[0102] The voice conversion system may further include a network analysis unit that analyzes the user's social network. The network analysis unit analyzes the user's relationships with friends, family, and colleagues, and adjusts the content of the voice conversion. For example, if the user is talking to family members, the system may convert the voice in a friendly tone. If the user is talking to colleagues in a business setting, the system may convert the voice in a formal tone. Furthermore, if the user is having a casual conversation with friends, the system may convert the voice in a relaxed tone. This allows for appropriate voice conversion according to the user's social network.

[0103] The voice conversion system can further estimate the user's emotions and adjust the tone and speed of the voice conversion based on the estimated emotions. For example, if the user is relaxed, the system can convert the voice at a slow speed with a soft tone. If the user is nervous, the system can convert the voice with a calm tone to reassure the user. If the user is excited, the system can convert the voice with an energetic tone to provide an expression that matches the user's emotions. This enables natural voice conversion according to the user's emotions.

[0104] The voice conversion system may further include a conversation history analysis unit that analyzes the user's past conversation history. The conversation history analysis unit analyzes the user's past conversation content and improves the accuracy of voice conversion. For example, if the user has frequently used a specific phrase in the past, the system may preferentially convert that phrase. Also, if the user has spoken a lot about a specific topic in the past, the system may preferentially convert terms related to that topic. Furthermore, based on the user's past conversation history, the system may learn the user's speaking patterns and provide more natural voice conversion. This enables highly accurate voice conversion based on the user's past conversation history.

[0105] The speech conversion system may further include a feedback collection unit that collects real-time feedback from the user. The feedback collection unit allows the user to provide real-time feedback on the results of speech conversion. For example, if the user is satisfied with the converted speech, the system may optimize the conversion algorithm based on the feedback. Also, if the user is dissatisfied with the conversion result, the system may identify improvements based on the feedback and incorporate them into the next conversion. Furthermore, if the user provides specific feedback on the conversion of a particular phrase or term, the system may improve the conversion accuracy based on the feedback. This allows flexible speech conversion based on real-time user feedback.

[0106] The voice conversion system can further estimate the user's emotions and customize the contents of the voice conversion based on the estimated emotions. For example, if the user is sad, the system can convert the voice to a comforting tone. If the user is happy, the system can convert the voice to a bright tone to further enhance the user's emotions. If the user is angry, the system can convert the voice to a calm tone to calm the user. This enables appropriate voice conversion according to the user's emotions.

[0107] The speech conversion system can further customize the contents of the speech conversion taking into account the geographic location information of the user. For example, if the user is in a particular area, phrases and terms related to the culture and customs of that area can be preferentially converted. Also, if the user is traveling, information related to tourist attractions and famous places of the visited area can be preferentially converted. Furthermore, if the user is on a business trip, speech related to the business customs and technical terms of the visited area can be preferentially converted. This enables appropriate speech conversion according to the geographic location information of the user.

[0108] The process flow of the second embodiment will be briefly described below.

[0109] Step 1: The voice collection unit collects the user's voice. The user's voice includes speaking characteristics and voice quality. The voice collection unit collects the user's voice using a microphone, and can also analyze the user's voice quality and speaking characteristics. For example, the voice collection unit analyzes the user's voice quality based on frequency characteristics and timbre. Step 2: The voice selection unit selects a voice from the voice data storage that is closest to the user's voice quality based on the voice data collected by the voice collection unit. Voices with various voice qualities are stored in the voice data storage, and the voice selection unit selects the voice that is closest to the user's voice quality. For example, for a user with a low-pitched voice quality, a low-pitched voice is selected. Step 3: The speech conversion unit uses a generation AI to convert the speech selected by the speech selection unit into another language. The generation AI uses a text generation AI (e.g., LLM) to convert the speech into another language, preserving the user's voice quality and speaking characteristics. For example, if the user says "Hello," the speech is converted into "Hello." Step 4: The audio transmission unit transmits the audio converted by the audio conversion unit to the earphones. The audio transmission unit transmits the converted audio to the earphones using wireless communication technology such as Bluetooth or Wi-Fi. This allows the user to listen to audio in other languages ​​in real time.

[0110] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0111] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) are input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may also be an AI agent. In addition, when the processes of each of the above-mentioned parts are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples.In addition, processing performed by AI including the generation AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI including the generation AI.

[0112] Furthermore, the above-described processing by the data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or an external device.

[0113] Each of the multiple elements including the above-mentioned voice collection unit, voice selection unit, voice conversion unit, and voice transmission unit is realized, for example, in at least one of the smart device 14 and the data processing device 12. For example, the voice collection unit collects the user's voice using the microphone 38B of the smart device 14. The voice selection unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and selects a voice similar to the user's voice quality from the voice data storage. The voice conversion unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and converts the voice into another language using a generation AI. The voice transmission unit transmits the converted voice to the earphones, for example, using the communication I / F 44 of the smart device 14. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0114] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0115] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0116] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0117] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0118] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0119] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0120] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0121] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0122] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0123] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0124] In the smart glasses 214, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0125] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0126] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0127] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0128] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device.

[0129] Each of the multiple elements including the above-mentioned voice collection unit, voice selection unit, voice conversion unit, and voice transmission unit is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the voice collection unit collects the user's voice using the microphone 238 of the smart glasses 214. The voice selection unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and selects a voice similar to the user's voice quality from the voice data storage. The voice conversion unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and converts the voice into another language using a generation AI. The voice transmission unit transmits the converted voice to the earphones, for example, by using the communication I / F 44 of the smart glasses 214. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0130] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0131] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0132] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0133] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0134] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0135] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0136] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0137] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0138] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0139] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0140] In the headset type terminal 314, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50, and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0141] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0142] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0143] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0144] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device.

[0145] Each of the multiple elements including the above-mentioned voice collection unit, voice selection unit, voice conversion unit, and voice transmission unit is realized, for example, in at least one of the headset type terminal 314 and the data processing device 12. For example, the voice collection unit collects the user's voice using the microphone 238 of the headset type terminal 314. The voice selection unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and selects a voice similar to the user's voice quality from the voice data storage. The voice conversion unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and converts the voice into another language using a generation AI. The voice transmission unit transmits the converted voice to the earphones, for example, using the communication I / F 44 of the headset type terminal 314. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0146] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0147] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0148] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0149] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0150] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0151] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0152] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0153] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and legs. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, and legs. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0154] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0155] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0156] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0157] In the robot 414, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads out the specific program 60 from the storage 50, and executes the specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0158] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0159] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0160] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0161] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0162] Each of the multiple elements including the above-mentioned voice collection unit, voice selection unit, voice conversion unit, and voice transmission unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the voice collection unit collects the user's voice using the microphone 238 of the robot 414. The voice selection unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and selects a voice similar to the user's voice quality from the voice data storage. The voice conversion unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and converts the voice into another language using a generation AI. The voice transmission unit transmits the converted voice to an earphone, for example, by using the communication I / F 44 of the robot 414. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0163] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[0164] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0165] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0166] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[0167] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0168] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0169] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0170] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0171] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0172] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0173] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0174] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0175] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0176] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0177] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0178] In the above example, the first to fourth embodiments are described separately, but some or all of these embodiments may be combined. The smart device 14, the smart glasses 214, the headset terminal 314, and the robot 414 are only examples, and they may be combined with each other, or may be other devices. In the above example, the first and second embodiments are described separately, but they may be combined.

[0179] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replaced with respect to the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the description and illustrations shown above omit explanations of technical common sense that do not require particular explanation in enabling the implementation of the technology of the present disclosure.

[0180] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0181] (Appendix 1) A voice collecting unit that collects a user's voice; a voice selection unit that selects a voice similar to a user's voice quality from a voice data storage based on the voice data collected by the voice collection unit; A voice conversion unit that converts the voice selected by the voice selection unit into another language by a generation AI; a sound transmission unit that transmits the sound converted by the sound conversion unit to an earphone; A system characterized by: (Appendix 2) The voice collecting unit is Analyze the user's voice quality and speaking characteristics 2. The system of claim 1, (Appendix 3) The audio selection unit is Select a voice that is close to the user's voice from the voice data storage 2. The system of claim 1, (Appendix 4) The voice conversion unit is Converting speech into other languages ​​using generative AI 2. The system of claim 1, (Appendix 5) The audio transmission unit is Transmit the converted audio to the earphones 2. The system of claim 1, (Appendix 6) The voice collecting unit is Describe a specific method for estimating a user's emotion and adjusting the timing of voice collection based on the estimated user's emotion. 2. The system of claim 1, (Appendix 7) The voice collecting unit is Analyze the user's past voice data and select the appropriate collection method 2. The system of claim 1, (Appendix 8) The voice collecting unit is During audio collection, the user's current environmental sounds are filtered to collect clear audio. 2. The system of claim 1, (Appendix 9) The voice collecting unit is We describe a specific method for estimating a user's emotion and prioritizing collected voice based on the estimated user's emotion. 2. The system of claim 1, (Appendix 10) The voice collecting unit is When collecting audio, the system takes into account the user's geographic location information to prioritize collection of audio that is highly relevant. 2. The system of claim 1, (Appendix 11) The voice collecting unit is When collecting audio, analyze the user's social media activity and collect related audio. 2. The system of claim 1, (Appendix 12) The audio selection unit is A specific method for estimating a user's emotion and adjusting the criteria for selecting a voice based on the estimated user's emotion is described. 2. The system of claim 1, (Appendix 13) The audio selection unit is When selecting a voice, the appropriate voice is selected by referring to the user's past voice data. 2. The system of claim 1, (Appendix 14) The audio selection unit is When selecting a voice, the selection is made taking into account the fluctuations in the user's current voice quality. 2. The system of claim 1, (Appendix 15) The audio selection unit is Estimating a user's emotion and determining a priority order of voices to be selected based on the estimated user's emotion 2. The system of claim 1, (Appendix 16) The audio selection unit is When selecting a voice, the system takes into account the user's geographic location to prioritize the most relevant voice. 2. The system of claim 1, (Appendix 17) The audio selection unit is When selecting a voice, analyze the user's social media activity and select the relevant voice. 2. The system of claim 1, (Appendix 18) The voice conversion unit is We describe a specific method for estimating a user's emotion and adjusting the expression method of the converted voice based on the estimated user's emotion. 2. The system of claim 1, (Appendix 19) The voice conversion unit is When converting voice, the appropriate conversion method is selected by referring to the user's past voice data. 2. The system of claim 1, (Appendix 20) The voice conversion unit is When converting voice, the conversion is performed taking into account the fluctuations in the user's current voice quality. 2. The system of claim 1, (Appendix 21) The voice conversion unit is Estimate the user's emotion and prioritize the speech to be converted based on the estimated user's emotion. 2. The system of claim 1, (Appendix 22) The voice conversion unit is When converting voice, the system takes into account the user's geographic location and prioritizes the most relevant voice. 2. The system of claim 1, (Appendix 23) The voice conversion unit is During voice conversion, analyze the user's social media activity and convert relevant voices. 2. The system of claim 1, (Appendix 24) The audio transmission unit is Describe a specific method for estimating a user's emotion and adjusting the method of voice transmission based on the estimated user's emotion. 2. The system of claim 1, (Appendix 25) The audio transmission unit is When transmitting voice, the appropriate transmission method is selected by referring to the user's past voice data. 2. The system of claim 1, (Appendix 26) The audio transmission unit is When transmitting voice, the user's current ambient sound is filtered to transmit clear voice. 2. The system of claim 1, (Appendix 27) The audio transmission unit is To estimate a user's emotion and determine a priority of speech transmission based on the estimated user's emotion. 2. The system of claim 1, (Appendix 28) The audio transmission unit is When transmitting voice, the system takes into account the user's geographic location information and prioritizes the transmission of voice that is more relevant. 2. The system of claim 1, (Appendix 29) The audio transmission unit is When a voice message is sent, the user's social media activity is analyzed and relevant voice messages are sent. 2. The system of claim 1, [Explanation of symbols]

[0182] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. A voice collecting unit that collects a user's voice; a voice selection unit that selects a voice similar to a user's voice quality from a voice data storage based on the voice data collected by the voice collection unit; A voice conversion unit that converts the voice selected by the voice selection unit into another language using a generation AI; a sound transmission unit that transmits the sound converted by the sound conversion unit to an earphone; A system characterized by:

2. The voice collecting unit is Analyze the user's voice quality and speaking characteristics 2. The system of claim 1 .

3. The audio selection unit is Select a voice that is close to the user's voice from the voice data storage 2. The system of claim 1 .

4. The voice conversion unit is Converting speech into other languages ​​using generative AI 2. The system of claim 1 .

5. The voice collecting unit is Describe a specific method for estimating a user's emotion and adjusting the timing of voice collection based on the estimated user's emotion.

2. The system of claim 1 .

6. The voice collecting unit is Analyze the user's past voice data and select the appropriate collection method 2. The system of claim 1 .

7. The voice collecting unit is During audio collection, the user's current environmental sounds are filtered to collect clear audio.

2. The system of claim 1 .

Images