System

Abstract

To provide a system that performs translation into another language while maintaining voice characteristics.SOLUTION: A system according to the embodiment comprises a reception unit, an extraction unit, and a generation unit. The reception unit inputs voice. The extraction unit extracts features of the voice input by the reception unit. The generation unit generates new voice based on translated text, on the basis of the features extracted by the extraction unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques have the problem that it is difficult to translate speech into another language while preserving its characteristics.

[0005] The system according to the embodiment aims to translate speech into another language while preserving its characteristics. [Means for solving the problem]

[0006] The system according to the embodiment includes a reception unit, an extraction unit, and a generation unit. The reception unit inputs a speech. The extraction unit extracts features of the speech input by the reception unit. The generation unit generates new speech based on the translated text, based on the features extracted by the extraction unit. Effect of the Invention

[0007] The system according to the embodiment can translate speech into another language while preserving its characteristics. [Brief description of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). Display 40A displays visible information such as text and images according to instructions from processor 46. Speaker 40B outputs voice according to instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Also, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, a home appliance, etc.) owned by a user. Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A speech translation system according to an embodiment of the present invention is a system that generates the voice of a voice actor in a movie or an anime as it is into another language. In this speech translation system, a voice actor speaks in Japanese, and a generation AI analyzes the voice and translates it into English while maintaining the voice actor's voice quality. The generation AI extracts the characteristics of the voice and generates a new voice based on the translated text. This new voice is a voice that speaks in English while maintaining the original voice actor's voice quality. This allows viewers of movies and anime to watch content in different languages ​​while enjoying the voice actor's voice quality. For example, a voice of a voice actor speaking in Japanese is input. At this time, the voice is recorded using a high-quality microphone. For example, a voice of a voice actor speaking a line in a movie scene is recorded. This voice is input into the generation AI. Next, the generation AI analyzes the input voice. The generation AI extracts the characteristics of the voice and identifies the voice actor's voice quality. For example, the characteristics such as the pitch, timbre, and rhythm of the voice are analyzed. This makes it possible to prepare the voice to be translated into another language while maintaining the voice actor's voice quality. The generation AI generates new speech based on the translated text based on the analyzed speech features. For example, the Japanese line "Hello" is translated into English "Hello," and based on the translated text, an English speech is generated while retaining the voice actor's voice quality. In this case, the generation AI uses voice synthesis technology to reproduce the voice actor's voice quality. The generated speech is a speech that speaks English while retaining the original voice actor's voice quality. This allows viewers to watch content in different languages ​​while enjoying the voice actor's voice quality. For example, it is possible to watch movies and anime that are spoken in English while keeping the voice quality of the voice actor who speaks Japanese. This technology improves the viewing experience of movies and anime and can accommodate viewers of different languages. For example, when an English-speaking viewer watches a Japanese anime, they can understand the content in English while enjoying the voice actor's voice quality. In addition, by retaining the voice actor's voice quality, it is possible to watch in a different language without losing the atmosphere of the work or the individuality of the character. This allows the speech translation system to generate the voice of a voice actor in a movie or anime in another language as it is.

[0029] The speech translation system according to the embodiment includes a reception unit, an extraction unit, and a generation unit. The reception unit inputs a voice spoken by a voice actor in Japanese. For example, the reception unit can input a voice recorded using a high-quality microphone. The reception unit can also directly accept voice data input in digital format. Furthermore, the reception unit has a function of uploading a voice file. For example, the reception unit can record a voice in which a voice actor speaks a line in a movie scene and input the voice data. The extraction unit extracts features of the voice input by the reception unit. For example, the extraction unit analyzes features such as pitch, timbre, and rhythm. The extraction unit can extract the features of the voice in detail using a voice analysis technology. For example, the extraction unit performs a spectrum analysis of the voice and extracts features of pitch and timbre. The extraction unit can also analyze the rhythm pattern of the voice and extract features of rhythm. The generation unit generates a new voice based on the translated text based on the features extracted by the extraction unit. The generation unit uses a generation AI to reproduce the voice quality using a voice synthesis technology. For example, the generation unit translates the Japanese phrase "Hello" into English "Hello" and generates English speech based on the translated text while maintaining the voice quality of the voice actor. The generation unit can also generate new speech using a speech database. For example, the generation unit generates new speech based on speech data recorded in the speech database. The generation unit can also generate speech using a machine learning model. For example, the generation unit generates speech using a neural network. Thereby, the speech translation system according to the embodiment can input a speech spoken by a voice actor in Japanese, extract its features, and generate new speech based on the translated text. A part or all of the above-mentioned processing in the generation unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the generation unit can generate speech using a generation AI model that inputs the features of the speech extracted by the extraction unit and the translated text, and outputs a new speech.

[0030] The reception unit inputs the voice of the voice actor speaking in Japanese. For example, the reception unit can input the voice recorded using a high-quality microphone. Specifically, it is possible to record clear and low-noise voice using a condenser microphone or a dynamic microphone used in a studio environment. The reception unit can also directly accept voice data input in digital format. For example, it supports common audio file formats such as WAV and MP3, and voice data can be input by uploading these files to the system. Furthermore, the reception unit has a function for uploading audio files. For example, it is possible to record the voice of a voice actor speaking lines in a movie scene and input the voice data. This allows the reception unit to accept voice data in various formats and methods, enabling flexible operation. Furthermore, the reception unit also has a function for accepting voice in real time, and can be used in scenarios such as live distribution and online meetings. For example, it is possible to receive voice streamed over the Internet in real time and immediately start translation processing. This allows the reception unit to handle not only recorded voice data but also real-time voice data, enabling it to be used for a wide range of purposes.

[0031] The extraction unit extracts the features of the voice input by the reception unit. For example, the extraction unit analyzes features such as the pitch, tone, and rhythm of the voice. Specifically, the extraction unit uses a voice analysis technology to perform a spectrum analysis of the voice and extracts the features of the pitch and tone of the voice. In the spectrum analysis, the voice signal is decomposed into frequency components and the intensity of each frequency component is analyzed, so that the features of the pitch and tone of the voice can be grasped in detail. The extraction unit can also analyze the rhythm pattern of the voice and extract the features of the rhythm. A method of analyzing the temporal fluctuation of the voice signal is used to analyze the rhythm pattern, and the features of the tempo and accent of the voice can be extracted. Furthermore, the extraction unit can perform a waveform analysis of the voice and analyze the energy distribution and amplitude fluctuation of the voice. This allows the extraction unit to analyze various features of the voice in detail and provide necessary information to the generation unit, which is the next processing step. The extraction unit can also extract the features of the voice using a machine learning algorithm. For example, by using a voice feature extraction model using deep learning, more accurate feature extraction is possible. This enables the extraction unit to capture subtle features and nuances of the voice, thereby improving the accuracy of voice synthesis in the generation unit.

[0032] The generation unit generates a new voice based on the translated text based on the features extracted by the extraction unit. The generation unit uses a generation AI to reproduce the voice quality using voice synthesis technology. For example, the generation unit translates the Japanese line "Hello" into English "Hello" and generates an English voice based on the translated text while retaining the voice quality of the voice actor. Specifically, the generation AI uses a voice synthesis model using a neural network to generate a new voice using the extracted voice features and the translated text as input. This voice synthesis model is trained with a large amount of voice data in advance and can reproduce the nuances of voice quality and pronunciation with high accuracy. The generation unit can also generate a new voice using a voice database. For example, the generation unit generates a new voice based on voice data recorded in the voice database. The voice database contains voice data of various voice qualities and pronunciations, and these data can be combined to generate a new voice. The generation unit can also generate a voice using a machine learning model. For example, the generation unit generates a voice using a neural network. As a result, the generation unit can generate speech using a generative AI model that takes the speech features extracted by the extraction unit and the translated text as input and outputs new speech. The generation unit also has a function of evaluating the quality of the generated speech and correcting or regenerating it as necessary. For example, if the generated speech is unnatural or the voice quality does not match, the generation unit can generate the speech again to improve the quality. As a result, the generation unit can generate high-quality, natural speech and provide it to the user.

[0033] The extraction unit can extract the pitch, timbre, and rhythm features of the voice. The extraction unit, for example, analyzes the pitch of the voice. For example, the extraction unit can analyze the frequency spectrum of the voice and identify the pitch of the voice. The extraction unit can also perform spectrum analysis of the voice to analyze the timbre. For example, the extraction unit can analyze the spectrum of the voice and extract the timbre features. Furthermore, the extraction unit can analyze the rhythm pattern of the voice to analyze the rhythm features. For example, the extraction unit can analyze the rhythm pattern of the voice and extract the rhythm features. This allows the extraction unit to extract the voice features in detail. A part or all of the above-mentioned processing in the extraction unit may be performed using, for example, AI, or may be performed without using AI. For example, the extraction unit can input the frequency spectrum of the voice to AI and cause AI to execute processing to identify the pitch of the voice.

[0034] The generation unit can reproduce the voice quality using a voice synthesis technology. The generation unit reproduces the voice quality using, for example, a text-to-speech synthesis (TTS) technology. For example, the generation unit can generate a new voice based on the translated text using a text-to-speech synthesis technology. The generation unit can also reproduce the voice quality using a waveform generation technology. For example, the generation unit can generate a voice waveform and reproduce the original voice quality. Furthermore, the generation unit can also reproduce the voice quality using a deep learning technology. For example, the generation unit can generate a voice using a neural network and reproduce the original voice quality. Thereby, the generation unit can reproduce the original voice quality using a voice synthesis technology. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the translated text to the generation AI and cause the generation AI to execute a process of generating a voice.

[0035] The generation unit can generate a new voice using the voice database. The generation unit generates a new voice based on, for example, voice data recorded in the voice database. For example, the generation unit can generate a new voice using voice data of a voice actor recorded in the voice database. The generation unit can also generate a new voice by combining multiple voice data recorded in the voice database. For example, the generation unit can generate a more natural voice by combining multiple voice data. Furthermore, the generation unit can learn the characteristics of the voice based on the voice data recorded in the voice database and generate a new voice. For example, the generation unit can learn the characteristics of the voice using the voice data recorded in the voice database and generate a new voice based on the characteristics. This allows the generation unit to generate a new voice using the voice database. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the voice data recorded in the voice database to the generation AI and cause the generation AI to execute a process of generating a new voice.

[0036] The generation unit can generate voice using a machine learning model. The generation unit can generate voice using, for example, a neural network. For example, the generation unit can generate voice using deep learning technology. The generation unit can also generate voice using a support vector machine (SVM). For example, the generation unit can learn voice features using a support vector machine to generate new voice. Furthermore, the generation unit can also generate voice using a generative inverse network (GAN). For example, the generation unit can generate more natural voice using a generative inverse network. This allows the generation unit to generate voice using a machine learning model. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input a neural network to the generation AI and cause the generation AI to execute a process of generating voice.

[0037] The reception unit can add a filtering function that automatically removes background noise when voice is input. The reception unit, for example, detects and filters ambient noise in real time when voice is input. For example, the reception unit can automatically remove noise in a specific frequency band when voice is input. The reception unit can also filter sounds other than the user's voice to obtain clear voice when voice is input. For example, the reception unit can detect surrounding environmental sounds when voice is input and remove them using noise canceling technology. This allows the reception unit to remove background noise and obtain clear voice. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input noise data detected when voice is input to the generation AI and cause the generation AI to execute noise removal processing.

[0038] The reception unit can add a function of adjusting the tone and tempo of the user's voice in real time when the voice is input. For example, if the user's voice is too high, the reception unit inputs the voice with a lowered tone. For example, the reception unit can analyze the frequency of the user's voice when inputting the voice and adjust the tone. In addition, if the user's voice is too fast, the reception unit can also input the voice with a slower tempo. For example, the reception unit can analyze the speed of the user's voice when inputting the voice and adjust the tempo. In addition, if the user's voice is too low, the reception unit can also input the voice with a higher tone. For example, the reception unit can analyze the frequency of the user's voice when inputting the voice and adjust the tone. Thereby, the reception unit can adjust the tone and tempo of the user's voice and input it. A part or all of the above-mentioned processing in the reception unit may be performed, for example, using AI, or may be performed without using AI. For example, the reception unit can input the voice frequency data analyzed at the time of voice input to the generation AI and cause the generation AI to adjust the tone and tempo.

[0039] The reception unit can preferentially input a highly relevant voice based on the geographical location information of the user when inputting a voice. For example, when the user is in a specific area, the reception unit preferentially inputs a voice related to the area. For example, the reception unit can acquire the geographical location information of the user and preferentially input a voice related to the area. In addition, when the user is traveling, the reception unit can also preferentially input a voice related to the travel destination. For example, the reception unit can acquire the geographical location information of the user and preferentially input a voice related to the travel destination. In addition, when the user is at home, the reception unit can also preferentially input a voice related to the home. For example, the reception unit can acquire the geographical location information of the user and preferentially input a voice related to the home. Thereby, the reception unit can preferentially input a highly relevant voice based on the geographical location information. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the geographical location information of the user to the generation AI and cause the generation AI to execute a process of preferentially inputting a highly relevant voice.

[0040] The reception unit can analyze the user's social media activity at the time of inputting the voice and input the related voice. The reception unit, for example, inputs voice related to a topic that the user has recently been talking about on social media. For example, the reception unit can analyze the content of the user's social media post and input the related voice preferentially. The reception unit can also input voice related to a topic in which the user's social media followers are interested. For example, the reception unit can analyze the interests of the user's social media followers and input the related voice preferentially. This allows the reception unit to input the related voice based on the social media activity. A part or all of the above-mentioned processing in the reception unit may be performed, for example, using AI or may be performed without using AI. For example, the reception unit can input the content of the user's social media post to the generation AI and cause the generation AI to execute a process of inputting the related voice.

[0041] The extraction unit can add a function of automatically correcting echo and reverb of the voice when extracting voice features. The extraction unit, for example, detects echo in real time when voice is input and corrects it. For example, the extraction unit can detect echo when voice is input and correct it using filtering technology. The extraction unit can also automatically detect reverb when voice is input and correct it. For example, the extraction unit can detect reverb when voice is input and correct it using filtering technology. Furthermore, the extraction unit can simultaneously correct echo and reverb when voice is input. For example, the extraction unit can detect echo and reverb when voice is input and correct them simultaneously using filtering technology. This allows the extraction unit to correct echo and reverb of voice to extract voice features. Part or all of the above-mentioned processing in the extraction unit may be performed using, for example, AI, or may be performed without using AI. For example, the extraction unit can input data of echo and reverb detected when voice is input to the generation AI and cause the generation AI to execute the correction processing.

[0042] The extraction unit can apply an algorithm that improves adaptability to different environmental sounds when extracting voice features. For example, when voice is input in a noisy environment, the extraction unit removes environmental sounds to extract voice features. For example, when voice is input in a noisy environment, the extraction unit can detect environmental sounds and remove them using a filtering technique. The extraction unit can also extract fine voice features when voice is input in a quiet environment. For example, when voice is input in a quiet environment, the extraction unit can detect fine voice features and extract them in detail. Furthermore, the extraction unit can apply an algorithm that adapts in real time to the fluctuating environmental sounds. For example, the extraction unit can detect the fluctuating environmental sounds in real time and extract the voice features using an adaptive algorithm. This allows the extraction unit to improve adaptability to different environmental sounds. A part or all of the above-mentioned processing in the extraction unit may be performed using, for example, AI, or may be performed without using AI. For example, the extraction unit can input data of the environmental sounds to the generation AI and cause the generation AI to execute extraction of the voice features using an adaptive algorithm.

[0043] The extraction unit can preferentially extract highly relevant features in consideration of the geographical location information of the user when extracting voice features. For example, when the user is in a specific area, the extraction unit preferentially extracts voice features related to that area. For example, the extraction unit can acquire the geographical location information of the user and preferentially extract voice features related to that area. In addition, when the user is traveling, the extraction unit can also preferentially extract voice features related to the travel destination. For example, the extraction unit can acquire the geographical location information of the user and preferentially extract voice features related to the travel destination. In addition, when the user is at home, the extraction unit can also preferentially extract voice features related to the home. For example, the extraction unit can acquire the geographical location information of the user and preferentially extract voice features related to the home. Thereby, the extraction unit can preferentially extract highly relevant voice features based on the geographical location information. Part or all of the above-mentioned processing in the extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the extraction unit can input the geographical location information of the user to the generation AI and cause the generation AI to execute a process of preferentially extracting highly relevant voice features.

[0044] The extraction unit can analyze the user's social media activities and extract related features when extracting voice features. The extraction unit, for example, extracts voice features related to topics that the user has been talking about recently on social media. For example, the extraction unit can analyze the content of the user's social media posts and preferentially extract related voice features. The extraction unit can also extract voice features related to topics in which the user's social media followers are interested. For example, the extraction unit can analyze the interests of the user's social media followers and preferentially extract related voice features. This allows the extraction unit to extract related voice features based on social media activities. A part or all of the above-mentioned processing in the extraction unit may be performed, for example, using AI or may be performed without using AI. For example, the extraction unit can input the content of the user's social media posts to the generation AI and cause the generation AI to execute a process of extracting related voice features.

[0045] The generation unit can add a function of reproducing subtle voice changes in order to improve the naturalness of the voice when generating the voice. For example, the generation unit reproduces subtle voice changes in real time when generating the voice. For example, the generation unit can naturally reproduce the intonation and rhythm of the voice when generating the voice. The generation unit can also adjust the volume of the voice in real time when generating the voice. For example, the generation unit can adjust the volume of the voice when generating the voice and generate a natural voice. This allows the generation unit to reproduce subtle voice changes in order to improve the naturalness of the voice. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI or may be performed without using a generation AI. For example, the generation unit can input voice data analyzed at the time of voice generation to the generation AI and cause the generation AI to reproduce subtle voice changes.

[0046] The generation unit can apply an algorithm that automatically corrects the difference in pronunciation between different languages ​​when generating speech. The generation unit automatically corrects the difference in pronunciation, for example, when translating from Japanese to English. For example, the generation unit can apply an algorithm that corrects Japanese pronunciation to English pronunciation. The generation unit can also automatically correct the difference in pronunciation when translating from English to French. For example, the generation unit can apply an algorithm that corrects English pronunciation to French pronunciation. Furthermore, the generation unit can automatically correct the difference in pronunciation when translating from Spanish to German. For example, the generation unit can apply an algorithm that corrects Spanish pronunciation to German pronunciation. This allows the generation unit to automatically correct the difference in pronunciation between different languages. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the translated text and the original speech data to the generation AI and cause the generation AI to execute a process of correcting the difference in pronunciation.

[0047] The generation unit can determine the priority order based on the submission time of the voice to be generated when generating the voice. For example, when the deadline is approaching, the generation unit generates the voice with priority. For example, the generation unit can generate the voice with priority for the voice with the upcoming submission time. In addition, the generation unit can prioritize the generation of other voices when the submission time is far away. For example, the generation unit can put off the voice with the distant submission time and generate other voices with priority. Furthermore, the generation unit can adjust the order of the voices to be generated based on the submission time. For example, the generation unit can adjust the order of the voices to be generated based on the submission time. Thereby, the generation unit can adjust the generation order of the voices based on the submission time. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input data of the submission time to the generation AI and cause the generation AI to execute a process of adjusting the generation order of the voices.

[0048] The generation unit can adjust the order based on the relevance of the generated voices when generating the voices. For example, the generation unit generates important voices with priority. For example, the generation unit can generate highly relevant voices with priority. The generation unit can also postpone low-relevance voices. For example, the generation unit can postpone low-relevance voices and generate other voices with priority. This allows the generation unit to generate highly relevant voices with priority. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input voice relevance data to the generation AI and cause the generation AI to execute a process of adjusting the generation order of the voices.

[0049] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0050] The reception unit can analyze the characteristics of the user's voice in real time and automatically adjust the tone and tempo of the voice. For example, if the user's voice is too high, the reception unit can input the voice with a lowered tone. Also, if the user's voice is too fast, the reception unit can input the voice with a slower tempo. Furthermore, if the user's voice is too low, the reception unit can input the voice with a higher tone. This allows the reception unit to adjust the tone and tempo of the user's voice and input it. A part or all of the above-mentioned processing in the reception unit may be performed using AI or may be performed without using AI. For example, the reception unit can input the voice frequency data analyzed at the time of voice input to the generation AI and cause the generation AI to adjust the tone and tempo.

[0051] The extraction unit can add a function of automatically correcting echo and reverb of the voice when extracting voice features. For example, the extraction unit can detect and correct echo in real time when voice is input. Also, the extraction unit can automatically detect and correct reverb when voice is input. Furthermore, the extraction unit can simultaneously correct echo and reverb when voice is input. This allows the extraction unit to extract voice features by correcting echo and reverb of the voice. A part or all of the above-mentioned processing in the extraction unit may be performed using AI or may be performed without using AI. For example, the extraction unit can input data of echo and reverb detected when voice is input to the generation AI and cause the generation AI to execute the correction processing.

[0052] The generation unit can add a function of reproducing subtle voice changes in order to improve the naturalness of the voice when generating the voice. For example, the generation unit can naturally reproduce the intonation and rhythm of the voice when generating the voice. In addition, the generation unit can adjust the volume of the voice in real time when generating the voice. This allows the generation unit to reproduce subtle voice changes in order to improve the naturalness of the voice. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input voice data analyzed at the time of voice generation to the generation AI and cause the generation AI to reproduce subtle voice changes.

[0053] The generation unit can apply an algorithm that automatically corrects the difference in pronunciation between different languages ​​when generating speech. For example, the difference in pronunciation can be automatically corrected when translating from Japanese to English. The difference in pronunciation can also be automatically corrected when translating from English to French. Furthermore, the difference in pronunciation can also be automatically corrected when translating from Spanish to German. This allows the generation unit to automatically correct the difference in pronunciation between different languages. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the translated text and the original speech data to the generation AI and cause the generation AI to execute a process of correcting the difference in pronunciation.

[0054] The generation unit can determine the priority order based on the submission time of the voice to be generated when generating the voice. For example, if the deadline is approaching, the generation of the voice can be prioritized. Also, if the submission time is far away, the generation of other voices can be prioritized. Furthermore, the order of the voices to be generated can be adjusted based on the submission time. This allows the generation unit to adjust the generation order of the voices based on the submission time. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input data on the submission time to the generation AI and cause the generation AI to execute a process of adjusting the generation order of the voices.

[0055] The process flow of the first embodiment will be briefly described below.

[0056] Step 1: The reception unit inputs the audio of a voice actor speaking in Japanese. For example, the reception unit can input audio recorded using a high-quality microphone. The reception unit can also directly accept audio data input in digital format. Furthermore, the reception unit has a function for uploading audio files. For example, the reception unit can record the audio of a voice actor speaking lines in a movie scene and input the audio data. Step 2: The extraction unit extracts features of the voice input by the reception unit. For example, the extraction unit analyzes features such as pitch, timbre, and rhythm. The extraction unit can extract detailed features of the voice using voice analysis technology. For example, the extraction unit performs a spectrum analysis of the voice to extract features of pitch and timbre. The extraction unit can also analyze the rhythm pattern of the voice to extract rhythm features. Step 3: The generation unit generates a new voice based on the translated text based on the features extracted by the extraction unit. The generation unit uses a generation AI to reproduce the voice quality using speech synthesis technology. For example, the generation unit translates the Japanese line "Kon'yō" into English "Hello" and generates an English voice based on the translated text while retaining the voice actor's voice quality. The generation unit can also generate a new voice using a voice database. For example, the generation unit generates a new voice based on the voice data recorded in the voice database. The generation unit can also generate a voice using a machine learning model. For example, the generation unit generates a voice using a neural network.

[0057] (Example 2) A speech translation system according to an embodiment of the present invention is a system that generates the voice of a voice actor in a movie or an anime as it is into another language. In this speech translation system, a voice actor speaks in Japanese, and a generation AI analyzes the voice and translates it into English while maintaining the voice actor's voice quality. The generation AI extracts the characteristics of the voice and generates a new voice based on the translated text. This new voice is a voice that speaks in English while maintaining the original voice actor's voice quality. This allows viewers of movies and anime to watch content in different languages ​​while enjoying the voice actor's voice quality. For example, a voice of a voice actor speaking in Japanese is input. At this time, the voice is recorded using a high-quality microphone. For example, a voice of a voice actor speaking a line in a movie scene is recorded. This voice is input into the generation AI. Next, the generation AI analyzes the input voice. The generation AI extracts the characteristics of the voice and identifies the voice actor's voice quality. For example, the characteristics such as the pitch, timbre, and rhythm of the voice are analyzed. This makes it possible to prepare the voice to be translated into another language while maintaining the voice actor's voice quality. The generation AI generates new speech based on the translated text based on the analyzed speech features. For example, the Japanese line "Hello" is translated into English "Hello," and based on the translated text, an English speech is generated while retaining the voice actor's voice quality. In this case, the generation AI uses voice synthesis technology to reproduce the voice actor's voice quality. The generated speech is a speech that speaks English while retaining the original voice actor's voice quality. This allows viewers to watch content in different languages ​​while enjoying the voice actor's voice quality. For example, it is possible to watch movies and anime that are spoken in English while keeping the voice quality of the voice actor who speaks Japanese. This technology improves the viewing experience of movies and anime and can accommodate viewers of different languages. For example, when an English-speaking viewer watches a Japanese anime, they can understand the content in English while enjoying the voice actor's voice quality. In addition, by retaining the voice actor's voice quality, it is possible to watch in a different language without losing the atmosphere of the work or the individuality of the character. This allows the speech translation system to generate the voice of a voice actor in a movie or anime in another language as it is.

[0058] The speech translation system according to the embodiment includes a reception unit, an extraction unit, and a generation unit. The reception unit inputs a voice spoken by a voice actor in Japanese. For example, the reception unit can input a voice recorded using a high-quality microphone. The reception unit can also directly accept voice data input in digital format. Furthermore, the reception unit has a function of uploading a voice file. For example, the reception unit can record a voice in which a voice actor speaks a line in a movie scene and input the voice data. The extraction unit extracts features of the voice input by the reception unit. For example, the extraction unit analyzes features such as pitch, timbre, and rhythm. The extraction unit can extract the features of the voice in detail using a voice analysis technology. For example, the extraction unit performs a spectrum analysis of the voice and extracts features of pitch and timbre. The extraction unit can also analyze the rhythm pattern of the voice and extract features of rhythm. The generation unit generates a new voice based on the translated text based on the features extracted by the extraction unit. The generation unit uses a generation AI to reproduce the voice quality using a voice synthesis technology. For example, the generation unit translates the Japanese phrase "Hello" into English "Hello" and generates English speech based on the translated text while maintaining the voice quality of the voice actor. The generation unit can also generate new speech using a speech database. For example, the generation unit generates new speech based on speech data recorded in the speech database. The generation unit can also generate speech using a machine learning model. For example, the generation unit generates speech using a neural network. Thereby, the speech translation system according to the embodiment can input a speech spoken by a voice actor in Japanese, extract its features, and generate new speech based on the translated text. A part or all of the above-mentioned processing in the generation unit may be performed, for example, using a generation AI, or may be performed without using a generation AI. For example, the generation unit can generate speech using a generation AI model that inputs the features of the speech extracted by the extraction unit and the translated text, and outputs a new speech.

[0059] The reception unit inputs the voice of the voice actor speaking in Japanese. For example, the reception unit can input the voice recorded using a high-quality microphone. Specifically, it is possible to record clear and low-noise voice using a condenser microphone or a dynamic microphone used in a studio environment. The reception unit can also directly accept voice data input in digital format. For example, it supports common audio file formats such as WAV and MP3, and voice data can be input by uploading these files to the system. Furthermore, the reception unit has a function for uploading audio files. For example, it is possible to record the voice of a voice actor speaking lines in a movie scene and input the voice data. This allows the reception unit to accept voice data in various formats and methods, enabling flexible operation. Furthermore, the reception unit also has a function for accepting voice in real time, and can be used in scenarios such as live distribution and online meetings. For example, it is possible to receive voice streamed over the Internet in real time and immediately start translation processing. This allows the reception unit to handle not only recorded voice data but also real-time voice data, enabling it to be used for a wide range of purposes.

[0060] The extraction unit extracts the features of the voice input by the reception unit. For example, the extraction unit analyzes features such as the pitch, tone, and rhythm of the voice. Specifically, the extraction unit uses a voice analysis technology to perform a spectrum analysis of the voice and extracts the features of the pitch and tone of the voice. In the spectrum analysis, the voice signal is decomposed into frequency components and the intensity of each frequency component is analyzed, so that the features of the pitch and tone of the voice can be grasped in detail. The extraction unit can also analyze the rhythm pattern of the voice and extract the features of the rhythm. A method of analyzing the temporal fluctuation of the voice signal is used to analyze the rhythm pattern, and the features of the tempo and accent of the voice can be extracted. Furthermore, the extraction unit can perform a waveform analysis of the voice and analyze the energy distribution and amplitude fluctuation of the voice. This allows the extraction unit to analyze various features of the voice in detail and provide necessary information to the generation unit, which is the next processing step. The extraction unit can also extract the features of the voice using a machine learning algorithm. For example, by using a voice feature extraction model using deep learning, more accurate feature extraction is possible. This enables the extraction unit to capture subtle features and nuances of the voice, thereby improving the accuracy of voice synthesis in the generation unit.

[0061] The generation unit generates a new voice based on the translated text based on the features extracted by the extraction unit. The generation unit uses a generation AI to reproduce the voice quality using voice synthesis technology. For example, the generation unit translates the Japanese line "Hello" into English "Hello" and generates an English voice based on the translated text while retaining the voice quality of the voice actor. Specifically, the generation AI uses a voice synthesis model using a neural network to generate a new voice using the extracted voice features and the translated text as input. This voice synthesis model is trained with a large amount of voice data in advance and can reproduce the nuances of voice quality and pronunciation with high accuracy. The generation unit can also generate a new voice using a voice database. For example, the generation unit generates a new voice based on voice data recorded in the voice database. The voice database contains voice data of various voice qualities and pronunciations, and these data can be combined to generate a new voice. The generation unit can also generate a voice using a machine learning model. For example, the generation unit generates a voice using a neural network. As a result, the generation unit can generate speech using a generative AI model that takes the speech features extracted by the extraction unit and the translated text as input and outputs new speech. The generation unit also has a function of evaluating the quality of the generated speech and correcting or regenerating it as necessary. For example, if the generated speech is unnatural or the voice quality does not match, the generation unit can generate the speech again to improve the quality. As a result, the generation unit can generate high-quality, natural speech and provide it to the user.

[0062] The extraction unit can extract the pitch, timbre, and rhythm features of the voice. The extraction unit, for example, analyzes the pitch of the voice. For example, the extraction unit can analyze the frequency spectrum of the voice and identify the pitch of the voice. The extraction unit can also perform spectrum analysis of the voice to analyze the timbre. For example, the extraction unit can analyze the spectrum of the voice and extract the timbre features. Furthermore, the extraction unit can analyze the rhythm pattern of the voice to analyze the rhythm features. For example, the extraction unit can analyze the rhythm pattern of the voice and extract the rhythm features. This allows the extraction unit to extract the voice features in detail. A part or all of the above-mentioned processing in the extraction unit may be performed using, for example, AI, or may be performed without using AI. For example, the extraction unit can input the frequency spectrum of the voice to AI and cause AI to execute processing to identify the pitch of the voice.

[0063] The generation unit can reproduce the voice quality using a voice synthesis technology. The generation unit reproduces the voice quality using, for example, a text-to-speech synthesis (TTS) technology. For example, the generation unit can generate a new voice based on the translated text using a text-to-speech synthesis technology. The generation unit can also reproduce the voice quality using a waveform generation technology. For example, the generation unit can generate a voice waveform and reproduce the original voice quality. Furthermore, the generation unit can also reproduce the voice quality using a deep learning technology. For example, the generation unit can generate a voice using a neural network and reproduce the original voice quality. Thereby, the generation unit can reproduce the original voice quality using a voice synthesis technology. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the translated text to the generation AI and cause the generation AI to execute a process of generating a voice.

[0064] The generation unit can generate a new voice using the voice database. The generation unit generates a new voice based on, for example, voice data recorded in the voice database. For example, the generation unit can generate a new voice using voice data of a voice actor recorded in the voice database. The generation unit can also generate a new voice by combining multiple voice data recorded in the voice database. For example, the generation unit can generate a more natural voice by combining multiple voice data. Furthermore, the generation unit can learn the characteristics of the voice based on the voice data recorded in the voice database and generate a new voice. For example, the generation unit can learn the characteristics of the voice using the voice data recorded in the voice database and generate a new voice based on the characteristics. This allows the generation unit to generate a new voice using the voice database. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the voice data recorded in the voice database to the generation AI and cause the generation AI to execute a process of generating a new voice.

[0065] The generation unit can generate voice using a machine learning model. The generation unit can generate voice using, for example, a neural network. For example, the generation unit can generate voice using deep learning technology. The generation unit can also generate voice using a support vector machine (SVM). For example, the generation unit can learn voice features using a support vector machine to generate new voice. Furthermore, the generation unit can also generate voice using a generative inverse network (GAN). For example, the generation unit can generate more natural voice using a generative inverse network. This allows the generation unit to generate voice using a machine learning model. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input a neural network to the generation AI and cause the generation AI to execute a process of generating voice.

[0066] The reception unit can estimate the user's emotion and adjust the voice input timing based on the estimated user's emotion. For example, the reception unit captures the user's facial expression with a camera and estimates the emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on the change in facial expression and adjusts the voice input timing. The reception unit can also record the user's voice and estimate the emotion using a voice analysis technology. For example, the reception unit analyzes the tone and speed of the voice, calculates an emotion score, and adjusts the voice input timing. The reception unit can also collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on the fluctuation of the heart rate and adjusts the voice input timing. This makes it possible to adjust the voice input timing according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit may input image data of a user taken by a camera to the generation AI and cause the generation AI to estimate the user's emotions.

[0067] The reception unit can add a filtering function that automatically removes background noise when voice is input. The reception unit, for example, detects and filters ambient noise in real time when voice is input. For example, the reception unit can automatically remove noise in a specific frequency band when voice is input. The reception unit can also filter sounds other than the user's voice to obtain clear voice when voice is input. For example, the reception unit can detect surrounding environmental sounds when voice is input and remove them using noise canceling technology. This allows the reception unit to remove background noise and obtain clear voice. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input noise data detected when voice is input to the generation AI and cause the generation AI to execute noise removal processing.

[0068] The reception unit can add a function of adjusting the tone and tempo of the user's voice in real time when the voice is input. For example, if the user's voice is too high, the reception unit inputs the voice with a lowered tone. For example, the reception unit can analyze the frequency of the user's voice when inputting the voice and adjust the tone. In addition, if the user's voice is too fast, the reception unit can also input the voice with a slower tempo. For example, the reception unit can analyze the speed of the user's voice when inputting the voice and adjust the tempo. In addition, if the user's voice is too low, the reception unit can also input the voice with a higher tone. For example, the reception unit can analyze the frequency of the user's voice when inputting the voice and adjust the tone. Thereby, the reception unit can adjust the tone and tempo of the user's voice and input it. A part or all of the above-mentioned processing in the reception unit may be performed, for example, using AI, or may be performed without using AI. For example, the reception unit can input the voice frequency data analyzed at the time of voice input to the generation AI and cause the generation AI to adjust the tone and tempo.

[0069] The reception unit can estimate the user's emotion and determine the priority of the voice to be input based on the estimated user's emotion. For example, the reception unit captures the user's facial expression with a camera and estimates the emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on the change in facial expression and determines the priority of the voice to be input. The reception unit can also record the user's voice and estimate the emotion using a voice analysis technology. For example, the reception unit analyzes the tone and speed of the voice, calculates an emotion score, and determines the priority of the voice to be input. The reception unit can also collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. For example, the reception unit calculates an emotion score based on the fluctuation of the heart rate and determines the priority of the voice to be input. This makes it possible to determine the priority of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, but is not limited to, a text generation AI (for example, LLM) or a multimodal generation AI. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit may input image data of a user taken by a camera to the generation AI and cause the generation AI to estimate the user's emotions.

[0070] The reception unit can preferentially input a highly relevant voice based on the geographical location information of the user when inputting a voice. For example, when the user is in a specific area, the reception unit preferentially inputs a voice related to the area. For example, the reception unit can acquire the geographical location information of the user and preferentially input a voice related to the area. In addition, when the user is traveling, the reception unit can also preferentially input a voice related to the travel destination. For example, the reception unit can acquire the geographical location information of the user and preferentially input a voice related to the travel destination. In addition, when the user is at home, the reception unit can also preferentially input a voice related to the home. For example, the reception unit can acquire the geographical location information of the user and preferentially input a voice related to the home. Thereby, the reception unit can preferentially input a highly relevant voice based on the geographical location information. A part or all of the above-mentioned processing in the reception unit may be performed using, for example, AI, or may be performed without using AI. For example, the reception unit can input the geographical location information of the user to the generation AI and cause the generation AI to execute a process of preferentially inputting a highly relevant voice.

[0071] The reception unit can analyze the user's social media activity at the time of inputting the voice and input the related voice. The reception unit, for example, inputs voice related to a topic that the user has recently been talking about on social media. For example, the reception unit can analyze the content of the user's social media post and input the related voice preferentially. The reception unit can also input voice related to a topic in which the user's social media followers are interested. For example, the reception unit can analyze the interests of the user's social media followers and input the related voice preferentially. This allows the reception unit to input the related voice based on the social media activity. A part or all of the above-mentioned processing in the reception unit may be performed, for example, using AI or may be performed without using AI. For example, the reception unit can input the content of the user's social media post to the generation AI and cause the generation AI to execute a process of inputting the related voice.

[0072] The extraction unit can estimate the user's emotion and adjust the voice feature extraction method based on the estimated user's emotion. For example, the extraction unit captures the user's facial expression with a camera and estimates the emotion using an emotion estimation algorithm. For example, the extraction unit calculates an emotion score based on the change in facial expression and adjusts the voice feature extraction method. The extraction unit can also record the user's voice and estimate the emotion using a voice analysis technique. For example, the extraction unit analyzes the tone and speed of the voice, calculates an emotion score, and adjusts the voice feature extraction method. The extraction unit can also collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. For example, the extraction unit calculates an emotion score based on the fluctuation of the heart rate and adjusts the voice feature extraction method. This makes it possible to adjust the voice feature extraction method according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the extraction unit may be performed using, for example, AI, or may be performed without using AI. For example, the extraction unit may input image data of a user taken by a camera to the generation AI and cause the generation AI to estimate the user's emotions.

[0073] The extraction unit can add a function of automatically correcting echo and reverb of the voice when extracting voice features. The extraction unit, for example, detects echo in real time when voice is input and corrects it. For example, the extraction unit can detect echo when voice is input and correct it using filtering technology. The extraction unit can also automatically detect reverb when voice is input and correct it. For example, the extraction unit can detect reverb when voice is input and correct it using filtering technology. Furthermore, the extraction unit can simultaneously correct echo and reverb when voice is input. For example, the extraction unit can detect echo and reverb when voice is input and correct them simultaneously using filtering technology. This allows the extraction unit to correct echo and reverb of voice to extract voice features. Part or all of the above-mentioned processing in the extraction unit may be performed using, for example, AI, or may be performed without using AI. For example, the extraction unit can input data of echo and reverb detected when voice is input to the generation AI and cause the generation AI to execute the correction processing.

[0074] The extraction unit can apply an algorithm that improves adaptability to different environmental sounds when extracting voice features. For example, when voice is input in a noisy environment, the extraction unit removes environmental sounds to extract voice features. For example, when voice is input in a noisy environment, the extraction unit can detect environmental sounds and remove them using a filtering technique. The extraction unit can also extract fine voice features when voice is input in a quiet environment. For example, when voice is input in a quiet environment, the extraction unit can detect fine voice features and extract them in detail. Furthermore, the extraction unit can apply an algorithm that adapts in real time to the fluctuating environmental sounds. For example, the extraction unit can detect the fluctuating environmental sounds in real time and extract the voice features using an adaptive algorithm. This allows the extraction unit to improve adaptability to different environmental sounds. A part or all of the above-mentioned processing in the extraction unit may be performed using, for example, AI, or may be performed without using AI. For example, the extraction unit can input data of the environmental sounds to the generation AI and cause the generation AI to execute extraction of the voice features using an adaptive algorithm.

[0075] The extraction unit can estimate the user's emotion and determine the priority of the voice features to be extracted based on the estimated user's emotion. For example, the extraction unit captures the user's facial expression with a camera and estimates the emotion using an emotion estimation algorithm. For example, the extraction unit calculates an emotion score based on the change in facial expression and determines the priority of the voice features to be extracted. The extraction unit can also record the user's voice and estimate the emotion using a voice analysis technology. For example, the extraction unit analyzes the tone and speed of the voice, calculates an emotion score, and determines the priority of the voice features to be extracted. The extraction unit can also collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. For example, the extraction unit calculates an emotion score based on the fluctuation of the heart rate and determines the priority of the voice features to be extracted. This makes it possible to determine the priority of the voice features according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the extraction unit may input image data of a user captured by a camera to the generation AI, and cause the generation AI to estimate the user's emotions.

[0076] The extraction unit can preferentially extract highly relevant features in consideration of the geographical location information of the user when extracting voice features. For example, when the user is in a specific area, the extraction unit preferentially extracts voice features related to that area. For example, the extraction unit can acquire the geographical location information of the user and preferentially extract voice features related to that area. In addition, when the user is traveling, the extraction unit can also preferentially extract voice features related to the travel destination. For example, the extraction unit can acquire the geographical location information of the user and preferentially extract voice features related to the travel destination. In addition, when the user is at home, the extraction unit can also preferentially extract voice features related to the home. For example, the extraction unit can acquire the geographical location information of the user and preferentially extract voice features related to the home. Thereby, the extraction unit can preferentially extract highly relevant voice features based on the geographical location information. Part or all of the above-mentioned processing in the extraction unit may be performed, for example, using AI, or may be performed without using AI. For example, the extraction unit can input the geographical location information of the user to the generation AI and cause the generation AI to execute a process of preferentially extracting highly relevant voice features.

[0077] The extraction unit can analyze the user's social media activities and extract related features when extracting voice features. The extraction unit, for example, extracts voice features related to topics that the user has been talking about recently on social media. For example, the extraction unit can analyze the content of the user's social media posts and preferentially extract related voice features. The extraction unit can also extract voice features related to topics in which the user's social media followers are interested. For example, the extraction unit can analyze the interests of the user's social media followers and preferentially extract related voice features. This allows the extraction unit to extract related voice features based on social media activities. A part or all of the above-mentioned processing in the extraction unit may be performed, for example, using AI or may be performed without using AI. For example, the extraction unit can input the content of the user's social media posts to the generation AI and cause the generation AI to execute a process of extracting related voice features.

[0078] The generation unit can estimate the user's emotion and adjust the expression method of the generated voice based on the estimated user's emotion. For example, the generation unit captures the user's facial expression with a camera and estimates the emotion using an emotion estimation algorithm. For example, the generation unit calculates an emotion score based on the change in facial expression and adjusts the expression method of the generated voice. The generation unit can also record the user's voice and estimate the emotion using a voice analysis technology. For example, the generation unit analyzes the tone and speed of the voice, calculates an emotion score, and adjusts the expression method of the generated voice. The generation unit can also collect the user's biodata (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. For example, the generation unit calculates an emotion score based on the fluctuation of the heart rate and adjusts the expression method of the generated voice. This makes it possible to adjust the expression method of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI may be a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. Some or all of the above-mentioned processing in the generation unit may be performed, for example, using AI, or may be performed without using AI. For example, the generation unit may input image data of a user captured by a camera to the generation AI, and cause the generation AI to estimate the user's emotions.

[0079] The generation unit can add a function of reproducing subtle voice changes in order to improve the naturalness of the voice when generating the voice. For example, the generation unit reproduces subtle voice changes in real time when generating the voice. For example, the generation unit can naturally reproduce the intonation and rhythm of the voice when generating the voice. The generation unit can also adjust the volume of the voice in real time when generating the voice. For example, the generation unit can adjust the volume of the voice when generating the voice and generate a natural voice. This allows the generation unit to reproduce subtle voice changes in order to improve the naturalness of the voice. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI or may be performed without using a generation AI. For example, the generation unit can input voice data analyzed at the time of voice generation to the generation AI and cause the generation AI to reproduce subtle voice changes.

[0080] The generation unit can apply an algorithm that automatically corrects the difference in pronunciation between different languages ​​when generating speech. The generation unit automatically corrects the difference in pronunciation, for example, when translating from Japanese to English. For example, the generation unit can apply an algorithm that corrects Japanese pronunciation to English pronunciation. The generation unit can also automatically correct the difference in pronunciation when translating from English to French. For example, the generation unit can apply an algorithm that corrects English pronunciation to French pronunciation. Furthermore, the generation unit can automatically correct the difference in pronunciation when translating from Spanish to German. For example, the generation unit can apply an algorithm that corrects Spanish pronunciation to German pronunciation. This allows the generation unit to automatically correct the difference in pronunciation between different languages. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the translated text and the original speech data to the generation AI and cause the generation AI to execute a process of correcting the difference in pronunciation.

[0081] The generation unit can estimate the user's emotion and adjust the length of the generated voice based on the estimated user's emotion. For example, the generation unit captures the user's facial expression with a camera and estimates the emotion using an emotion estimation algorithm. For example, the generation unit calculates an emotion score based on a change in facial expression and adjusts the length of the generated voice. The generation unit can also record the user's voice and estimate the emotion using a voice analysis technique. For example, the generation unit analyzes the tone and speed of the voice, calculates an emotion score, and adjusts the length of the generated voice. The generation unit can also collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. For example, the generation unit calculates an emotion score based on the fluctuation of the heart rate and adjusts the length of the generated voice. This makes it possible to adjust the length of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the generation unit may be performed using, for example, AI, or may be performed without using AI. For example, the generation unit may input image data of a user taken by a camera to the generation AI and cause the generation AI to estimate the user's emotions.

[0082] The generation unit can determine the priority order based on the submission time of the voice to be generated when generating the voice. For example, when the deadline is approaching, the generation unit generates the voice with priority. For example, the generation unit can generate the voice with priority for the voice with the upcoming submission time. In addition, the generation unit can prioritize the generation of other voices when the submission time is far away. For example, the generation unit can put off the voice with the distant submission time and generate other voices with priority. Furthermore, the generation unit can adjust the order of the voices to be generated based on the submission time. For example, the generation unit can adjust the order of the voices to be generated based on the submission time. Thereby, the generation unit can adjust the generation order of the voices based on the submission time. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input data of the submission time to the generation AI and cause the generation AI to execute a process of adjusting the generation order of the voices.

[0083] The generation unit can adjust the order based on the relevance of the generated voices when generating the voices. For example, the generation unit generates important voices with priority. For example, the generation unit can generate highly relevant voices with priority. The generation unit can also postpone low-relevance voices. For example, the generation unit can postpone low-relevance voices and generate other voices with priority. This allows the generation unit to generate highly relevant voices with priority. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input voice relevance data to the generation AI and cause the generation AI to execute a process of adjusting the generation order of the voices.

[0084] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0085] The reception unit can analyze the characteristics of the user's voice in real time and automatically adjust the tone and tempo of the voice. For example, if the user's voice is too high, the reception unit can input the voice with a lowered tone. Also, if the user's voice is too fast, the reception unit can input the voice with a slower tempo. Furthermore, if the user's voice is too low, the reception unit can input the voice with a higher tone. This allows the reception unit to adjust the tone and tempo of the user's voice and input it. A part or all of the above-mentioned processing in the reception unit may be performed using AI or may be performed without using AI. For example, the reception unit can input the voice frequency data analyzed at the time of voice input to the generation AI and cause the generation AI to adjust the tone and tempo.

[0086] The extraction unit can add a function of automatically correcting echo and reverb of the voice when extracting voice features. For example, the extraction unit can detect and correct echo in real time when voice is input. Also, the extraction unit can automatically detect and correct reverb when voice is input. Furthermore, the extraction unit can simultaneously correct echo and reverb when voice is input. This allows the extraction unit to extract voice features by correcting echo and reverb of the voice. A part or all of the above-mentioned processing in the extraction unit may be performed using AI or may be performed without using AI. For example, the extraction unit can input data of echo and reverb detected when voice is input to the generation AI and cause the generation AI to execute the correction processing.

[0087] The generation unit can add a function of reproducing subtle voice changes in order to improve the naturalness of the voice when generating the voice. For example, the generation unit can naturally reproduce the intonation and rhythm of the voice when generating the voice. In addition, the generation unit can adjust the volume of the voice in real time when generating the voice. This allows the generation unit to reproduce subtle voice changes in order to improve the naturalness of the voice. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input voice data analyzed at the time of voice generation to the generation AI and cause the generation AI to reproduce subtle voice changes.

[0088] The generation unit can apply an algorithm that automatically corrects the difference in pronunciation between different languages ​​when generating speech. For example, the difference in pronunciation can be automatically corrected when translating from Japanese to English. The difference in pronunciation can also be automatically corrected when translating from English to French. Furthermore, the difference in pronunciation can also be automatically corrected when translating from Spanish to German. This allows the generation unit to automatically correct the difference in pronunciation between different languages. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input the translated text and the original speech data to the generation AI and cause the generation AI to execute a process of correcting the difference in pronunciation.

[0089] The generation unit can determine the priority order based on the submission time of the voice to be generated when generating the voice. For example, if the deadline is approaching, the generation of the voice can be prioritized. Also, if the submission time is far away, the generation of other voices can be prioritized. Furthermore, the order of the voices to be generated can be adjusted based on the submission time. This allows the generation unit to adjust the generation order of the voices based on the submission time. A part or all of the above-mentioned processing in the generation unit may be performed using a generation AI, or may be performed without using a generation AI. For example, the generation unit can input data on the submission time to the generation AI and cause the generation AI to execute a process of adjusting the generation order of the voices.

[0090] The reception unit can estimate the user's emotion and adjust the input timing of the voice based on the estimated user's emotion. For example, the reception unit can capture the user's facial expression with a camera and estimate the emotion using an emotion estimation algorithm. The reception unit can also record the user's voice and estimate the emotion using a voice analysis technology. Furthermore, the reception unit can collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. This makes it possible to adjust the input timing of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using an emotion engine or a generation AI. The generation AI is, for example, a text generation AI or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the reception unit may be performed using an AI or may be performed without using an AI. For example, the reception unit can input image data of the user captured by a camera to the generation AI and cause the generation AI to execute the estimation of the user's emotion.

[0091] The reception unit can estimate the user's emotion and determine the priority of the voice to be input based on the estimated user's emotion. For example, the reception unit can capture the user's facial expression with a camera and estimate the emotion using an emotion estimation algorithm. The reception unit can also record the user's voice and estimate the emotion using a voice analysis technology. Furthermore, the user's biometric data (heart rate and electrodermal activity) can be collected with a sensor and the emotion can be estimated using an emotion estimation algorithm. This makes it possible to determine the priority of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using an emotion engine or a generation AI. The generation AI is, for example, a text generation AI or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the reception unit may be performed using an AI or may be performed without using an AI. For example, the reception unit can input image data of the user captured by a camera to the generation AI and cause the generation AI to execute the estimation of the user's emotion.

[0092] The extraction unit can estimate the user's emotion and adjust the voice feature extraction method based on the estimated user's emotion. For example, the extraction unit can capture the user's facial expression with a camera and estimate the emotion using an emotion estimation algorithm. The extraction unit can also record the user's voice and estimate the emotion using a voice analysis technology. Furthermore, the user's biometric data (heart rate and electrodermal activity) can be collected with a sensor and the emotion can be estimated using an emotion estimation algorithm. This makes it possible to adjust the voice feature extraction method according to the user's emotion. The emotion estimation is realized using an emotion estimation function using an emotion engine or a generation AI. The generation AI is, for example, a text generation AI or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the extraction unit may be performed using an AI or may be performed without using an AI. For example, the extraction unit can input image data of the user captured by a camera to the generation AI and cause the generation AI to execute the estimation of the user's emotion.

[0093] The generation unit can estimate the user's emotion and adjust the expression method of the generated voice based on the estimated user's emotion. For example, the generation unit can capture the user's facial expression with a camera and estimate the emotion using an emotion estimation algorithm. The generation unit can also record the user's voice and estimate the emotion using a voice analysis technology. Furthermore, the generation unit can collect the user's biometric data (heart rate and electrodermal activity) with a sensor and estimate the emotion using an emotion estimation algorithm. This makes it possible to adjust the expression method of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using an emotion engine or a generation AI. The generation AI is, for example, a text generation AI or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the generation unit may be performed using an AI or may be performed without using an AI. For example, the generation unit can input image data of the user captured by a camera to the generation AI and cause the generation AI to execute the estimation of the user's emotion.

[0094] The generation unit can estimate the user's emotion and adjust the length of the generated voice based on the estimated user's emotion. For example, the generation unit can capture the user's facial expression with a camera and estimate the emotion using an emotion estimation algorithm. The generation unit can also record the user's voice and estimate the emotion using a voice analysis technology. Furthermore, the user's biometric data (heart rate and electrodermal activity) can be collected with a sensor and the emotion can be estimated using an emotion estimation algorithm. This makes it possible to adjust the length of the voice according to the user's emotion. The emotion estimation is realized using an emotion estimation function using an emotion engine or a generation AI. The generation AI is, for example, a text generation AI or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the generation unit may be performed using an AI or may be performed without using an AI. For example, the generation unit can input image data of the user captured by a camera to the generation AI and cause the generation AI to execute the estimation of the user's emotion.

[0095] The process flow of the second embodiment will be briefly described below.

[0096] Step 1: The reception unit inputs the audio of a voice actor speaking in Japanese. For example, the reception unit can input audio recorded using a high-quality microphone. The reception unit can also directly accept audio data input in digital format. Furthermore, the reception unit has a function for uploading audio files. For example, the reception unit can record the audio of a voice actor speaking lines in a movie scene and input the audio data. Step 2: The extraction unit extracts features of the voice input by the reception unit. For example, the extraction unit analyzes features such as pitch, timbre, and rhythm. The extraction unit can extract detailed features of the voice using voice analysis technology. For example, the extraction unit performs a spectrum analysis of the voice to extract features of pitch and timbre. The extraction unit can also analyze the rhythm pattern of the voice to extract rhythm features. Step 3: The generation unit generates a new voice based on the translated text based on the features extracted by the extraction unit. The generation unit uses a generation AI to reproduce the voice quality using speech synthesis technology. For example, the generation unit translates the Japanese line "Kon'yō" into English "Hello" and generates an English voice based on the translated text while retaining the voice actor's voice quality. The generation unit can also generate a new voice using a voice database. For example, the generation unit generates a new voice based on the voice data recorded in the voice database. The generation unit can also generate a voice using a machine learning model. For example, the generation unit generates a voice using a neural network.

[0097] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0098] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) are input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may also be an AI agent. In addition, when the processes of each of the above-mentioned parts are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples.In addition, processing performed by AI including the generation AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI including the generation AI.

[0099] Furthermore, the above-described processing by the data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or an external device.

[0100] Each of the multiple elements including the above-mentioned reception unit, extraction unit, and generation unit is realized, for example, by at least one of the smart device 14 and the data processing device 12. For example, the reception unit can input the voice of a voice actor using the microphone 38B of the smart device 14. The reception unit can also receive digital voice data via the communication I / F 26 of the data processing device 12. The extraction unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes the characteristics of the voice. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and generates new voice based on the translated text. The generation unit may be realized, for example, by the control unit 46A of the smart device 14. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0101] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0102] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0103] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0104] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0105] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0106] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0107] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0108] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0109] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0110] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0111] In the smart glasses 214, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0112] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0113] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0114] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0115] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device.

[0116] Each of the multiple elements including the above-mentioned reception unit, extraction unit, and generation unit is realized, for example, by at least one of the smart glasses 214 and the data processing device 12. For example, the reception unit can input the voice of a voice actor using the microphone 238 of the smart glasses 214. The reception unit can also receive digital voice data via the communication I / F 26 of the data processing device 12. The extraction unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes the characteristics of the voice. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and generates new voice based on the translated text. The generation unit may be realized, for example, by the control unit 46A of the smart glasses 214. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0117] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0118] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0119] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0120] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0121] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0122] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0123] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0124] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0125] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0126] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0127] In the headset type terminal 314, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50, and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0128] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0129] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0130] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0131] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device.

[0132] Each of the multiple elements including the above-mentioned reception unit, extraction unit, and generation unit is realized, for example, by at least one of the headset type terminal 314 and the data processing device 12. For example, the reception unit can input the voice of a voice actor using the microphone 238 of the headset type terminal 314. The reception unit can also receive digital voice data via the communication I / F 26 of the data processing device 12. The extraction unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes the characteristics of the voice. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and generates new voice based on the translated text. The generation unit may be realized, for example, by the control unit 46A of the headset type terminal 314. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0133] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0134] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0135] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0136] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0137] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0138] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0139] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0140] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and legs. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, and legs. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0141] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0142] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0143] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0144] In the robot 414, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads out the specific program 60 from the storage 50, and executes the specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0145] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0146] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0147] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0148] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0149] Each of the multiple elements including the above-mentioned reception unit, extraction unit, and generation unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the reception unit can input the voice of a voice actor using the microphone 238 of the robot 414. The reception unit can also receive digital voice data via the communication I / F 26 of the data processing device 12. The extraction unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and analyzes the characteristics of the voice. The generation unit is realized, for example, by the specific processing unit 290 of the data processing device 12, and generates a new voice based on the translated text. The generation unit may be realized, for example, by the control unit 46A of the robot 414. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0150] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[0151] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0152] These emotions are distributed in the three o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0153] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).

[0154] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0155] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0156] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0157] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0158] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0159] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0160] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0161] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0162] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0163] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0164] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0165] In the above example, the first to fourth embodiments are described separately, but some or all of these embodiments may be combined. The smart device 14, the smart glasses 214, the headset terminal 314, and the robot 414 are only examples, and they may be combined with each other, or may be other devices. In the above example, the first and second embodiments are described separately, but they may be combined.

[0166] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replaced with respect to the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the description and illustrations shown above omit explanations of technical common sense that do not require particular explanation in enabling the implementation of the technology of the present disclosure.

[0167] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0168] (Appendix 1) A reception unit for inputting voice; an extraction unit that extracts features of the voice input by the reception unit; a generator for generating a new speech based on the translated text based on the features extracted by the extractor. A system characterized by: (Appendix 2) The extraction unit is Extracting vocal pitch, timbre, and rhythmic features 2. The system of claim 1, (Appendix 3) The generation unit is Uses voice synthesis technology to recreate voice quality 2. The system of claim 1, (Appendix 4) The generation unit is Generate new voices using a voice database 2. The system of claim 1, (Appendix 5) The generation unit is Generate audio using machine learning models 2. The system of claim 1, (Appendix 6) The reception unit is To estimate a user's emotion and adjust the timing of voice input based on the estimated user's emotion. 2. The system of claim 1, (Appendix 7) The reception unit is Add a filtering function that automatically removes background noise when inputting voice. 2. The system of claim 1, (Appendix 8) The reception unit is Add the ability to adjust the tone and tempo of a user's voice in real time as they input it. 2. The system of claim 1, (Appendix 9) The reception unit is Estimate the user's emotions and prioritize input voices based on the estimated user's emotions. 2. The system of claim 1, (Appendix 10) The reception unit is When entering voice input, prioritize relevant voice input based on the user's geographic location 2. The system of claim 1, (Appendix 11) The reception unit is When you input voice, it analyzes your social media activity and inputs relevant voice. 2. The system of claim 1, (Appendix 12) The extraction unit is Estimating a user's emotion and adjusting a speech feature extraction method based on the estimated user's emotion 2. The system of claim 1, (Appendix 13) The extraction unit is Add a function to automatically correct echo and reverb when extracting audio features. 2. The system of claim 1, (Appendix 14) The extraction unit is When extracting audio features, an algorithm is applied that improves adaptability to different environmental sounds. 2. The system of claim 1, (Appendix 15) The extraction unit is Estimate the user's emotion and prioritize the voice features to be extracted based on the estimated user's emotion. 2. The system of claim 1, (Appendix 16) The extraction unit is When extracting voice features, the system takes into account the user's geographic location information to prioritize the extraction of highly relevant features. 2. The system of claim 1, (Appendix 17) The extraction unit is When extracting voice features, we analyze the user's social media activity and extract relevant features. 2. The system of claim 1, (Appendix 18) The generation unit is Estimate the user's emotions and adjust the way the generated voice is expressed based on the estimated user's emotions. 2. The system of claim 1, (Appendix 19) The generation unit is Add a function to reproduce subtle voice changes in order to improve the naturalness of voice during voice generation. 2. The system of claim 1, (Appendix 20) The generation unit is Applying algorithms to automatically compensate for pronunciation differences between different languages ​​during speech generation 2. The system of claim 1, (Appendix 21) The generation unit is Estimate the user's emotion and adjust the length of the generated voice based on the estimated user's emotion. 2. The system of claim 1, (Appendix 22) The generation unit is During audio generation, priority is determined based on when the audio to be generated was submitted. 2. The system of claim 1, (Appendix 23) The generation unit is During speech generation, adjust the order based on the relevance of the generated speech 2. The system of claim 1, [Explanation of symbols]

[0169] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. A reception unit for inputting voice; an extraction unit that extracts features of the voice input by the reception unit; a generator for generating a new speech based on the translated text based on the features extracted by the extractor. A system characterized by:

2. The extraction unit is Extracting vocal pitch, timbre, and rhythmic features 2. The system of claim 1 .

3. The generation unit is Uses voice synthesis technology to recreate voice quality 2. The system of claim 1 .

4. The generation unit is Generate new voices using a voice database 2. The system of claim 1 .

5. The generation unit is Generate audio using machine learning models 2. The system of claim 1 .

6. The reception unit is To estimate a user's emotion and adjust the timing of voice input based on the estimated user's emotion.

2. The system of claim 1 .

7. The reception unit is Add a filtering function that automatically removes background noise when inputting voice.

2. The system of claim 1 .

8. The reception unit is Add the ability to adjust the tone and tempo of a user's voice in real time when inputting voice.

2. The system of claim 1 .

9. The reception unit is Estimate the user's emotions and prioritize input voices based on the estimated user's emotions.

2. The system of claim 1 .

10. The reception unit is When entering voice input, prioritize relevant voice input based on the user's geographic location 2. The system of claim 1 .

Images