System

Abstract

A system is provided.SOLUTION: A system comprising: audio input means; means for transmitting audio data; means for pre-processing the audio data; means for extracting audio features; means for selecting an avatar model based on the audio features; means for generating an image based on the avatar model; means for checking and adjusting the quality of the generated avatar; means for transmitting the generated avatar image data; means for displaying the avatar image; and means for receiving user feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As demand for virtual avatars grows, it is important to generate avatars that have voice and facial consistency. However, conventional technologies have difficulty selecting appropriate avatar models based on voice features and generating realistic images, and they tend to be expensive. Furthermore, if the quality of the generated avatars is insufficient, users may feel uncomfortable, reducing their usefulness. To solve this problem, a technology is needed to efficiently generate realistic and natural avatars from voice data. [Means for solving the problem]

[0005] The present invention provides a system including a voice input unit, a voice data transmission unit, a voice data preprocessing unit, a voice feature extraction unit, an avatar model selection unit based on the voice feature, an image generation unit based on the avatar model, a quality check and adjustment unit for the generated avatar, a transmission unit for the generated avatar image data, a display unit for the avatar image, and a user feedback receiving unit. This system utilizes voice features to efficiently generate natural-looking avatars, reducing costs and providing high-quality avatars. Furthermore, by performing quality checks and adjustments, it is possible to display avatars that do not appear strange to the user.

[0006] "Audio input means" refers to a device, function, or module for receiving audio data.

[0007] "Means for transmitting voice data" refers to a device, function, or module for sending collected voice data to another system or server.

[0008] "Means for pre-processing audio data" refers to a device, function, or module for removing noise from audio data or converting audio data into a form that is easier to analyze.

[0009] "Means for extracting audio features" refers to a device, function, or module for analyzing and extracting features such as pitch, formants, rhythm, and tone from audio data.

[0010] "Means for selecting an avatar model based on audio features" refers to a device, function, or module for selecting an appropriate avatar model (e.g., anime-style or live-action-style) based on extracted audio features.

[0011] "Means for generating an image based on an avatar model" refers to a device, function, or module for creating a specific avatar image based on a selected avatar model.

[0012] "Means for checking and adjusting the quality of the generated avatar" refers to a device, function, or module for checking whether the generated avatar image is of the desired quality and correcting any inappropriate parts.

[0013] "Means for transmitting generated avatar image data" refers to a device, function, or module for transmitting the generated avatar image data to a user's terminal.

[0014] "Means for displaying an avatar image" refers to a device, function, or module for displaying the generated avatar image for a user to view.

[0015] The "means for receiving user feedback" refers to a device, function, or module that allows a user to input opinions or requests regarding the generated avatar and communicate them to the system. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention provides a system for generating avatars from voice data. The system aims to generate natural and realistic avatar images from a user's voice input, and is implemented by the following method and procedure.

[0038] System Overview

[0039] The system includes the following major components:

[0040] Voice input means

[0041] A means of transmitting audio data

[0042] A means of preprocessing audio data

[0043] A means of extracting speech features

[0044] A method for selecting avatar models based on speech features

[0045] A method for generating images based on avatar models

[0046] A way to check and adjust the quality of the generated avatars

[0047] A means for transmitting the generated avatar image data

[0048] A means of displaying an avatar image

[0049] A way to receive user feedback

[0050] Program processing

[0051] The program processing in this system will be explained in natural language below.

[0052] 1. Audio input

[0053] The user speaks into the device, and can include the message they want to convey and the characteristics of the avatar they want to create. For example, they might say, "Hello, my name is Sato. Please create an anime character for me."

[0054] 2. Sending audio data

[0055] The terminal records the user's voice and transmits the digital voice data to the server.

[0056] 3. Preprocessing of audio data

[0057] The server receives the audio data and first performs noise reduction, which filters out background noise and unwanted sounds.

[0058] The audio data is then converted into a digital format that makes it easier to analyze.

[0059] 4. Extraction of speech features

[0060] The server extracts speech features from the speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0061] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[0062] 5. Select an avatar model

[0063] The server selects the most suitable avatar model based on the extracted voice features. For example, if the voice is bright and high-pitched, it selects a youthful anime character.

[0064] To select a model, a machine learning algorithm is used, referencing pre-trained voice-character pair data.

[0065] 6. Image Generation

[0066] The server generates an image based on the selected avatar model, specifically using deep learning models such as GAN (Generative Adversarial Network) and StyleGAN.

[0067] This generates a facial pattern corresponding to the audio features.

[0068] 7. Avatar quality check and adjustment

[0069] The server checks the quality of the generated avatar, checking for unnatural features such as facial distortion or unnatural facial expressions.

[0070] If necessary, fine-tune using post-processing algorithms.

[0071] 8. Sending and displaying avatar images

[0072] The server generates the final avatar image data and sends it to the user's terminal.

[0073] The device displays the received avatar image to the user, who can then review the avatar image and provide feedback if necessary.

[0074] Specific examples

[0075] Below are some specific usage examples.

[0076] Use case 1: Anime-style avatar generation

[0077] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create an anime character for me."

[0078] 2. The device records the audio and sends it to the server.

[0079] 3. The server preprocesses the audio data, removes noise, and extracts features.

[0080] 4. The server selects a youthful anime character model based on the features.

[0081] 5. The server uses GAN to generate an animated avatar.

[0082] 6. The server checks the quality of the generated avatar and makes adjustments if necessary.

[0083] 7. The server sends the avatar image to the device.

[0084] 8. The device displays the avatar to the user, who confirms it.

[0085] This system allows users to quickly generate a natural voice-based avatar to enhance visual communication.

[0086] The processing flow will be explained below.

[0087] Step 1:

[0088] The user speaks into the device's microphone, describing the desired avatar's characteristics and a message. For example, they might say, "Hello, I'm Sato. Please create an anime character for me."

[0089] Step 2:

[0090] The device records the user's voice and generates digital audio data that is stored in a common audio file format (e.g., WAV, MP3).

[0091] Step 3:

[0092] The device sends the recorded audio data to a server, using an internet connection for data transfer.

[0093] Step 4:

[0094] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[0095] Step 5:

[0096] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are extracted using FFT (Fast Fourier Transform) and cepstrum analysis.

[0097] Step 6:

[0098] The server selects an appropriate avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest, SVM) to select the best model from a pre-trained voice-character pair database.

[0099] Step 7:

[0100] The server generates an avatar image based on the selected avatar model, and uses a deep learning model (e.g., GAN, StyleGAN) to generate a facial pattern corresponding to the audio features.

[0101] Step 8:

[0102] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[0103] Step 9:

[0104] The server sends the final avatar image data to the device, where it is saved in high-resolution format and provided to the user.

[0105] Step 10:

[0106] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[0107] This series of steps allows users to quickly generate a natural, high-quality avatar based on their voice.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Conventional avatar generation systems have had many problems when generating natural and realistic avatars from user voice, particularly in the preprocessing of voice data, feature extraction, and final image generation and quality adjustment. Specifically, the efficiency of noise removal from voice data was low, and the accuracy of extracted features was insufficient, making it often impossible to select the optimal avatar model. Furthermore, the quality of the generated images was unstable, with many unnatural parts. A method to resolve these issues is needed.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes means for performing FFT analysis on the voice data, means for generating avatar images using GAN, and means for adjusting the quality of the generated images using a post-processing algorithm, which enables highly accurate analysis and noise removal of the voice data, selection of the optimal avatar model, and generation and adjustment of high-quality avatar images.

[0113] "Voice input means" refers to a device or interface that allows a user to input voice, such as a microphone or voice recognition software.

[0114] "Means for transmitting audio data" refers to the communications means for transmitting audio data from the user's device to the server, typically including an internet connection and the HTTP protocol.

[0115] "Means for pre-processing audio data" refers to processes for converting recorded audio data into an analyzable format, including, for example, noise reduction and digital conversion.

[0116] "Means for extracting speech features" refers to techniques for analyzing and extracting necessary speech features from preprocessed speech data, including FFT (Fast Fourier Transform) and cepstrum analysis.

[0117] "Means for selecting an avatar model based on speech features" refers to an algorithm for selecting an optimal avatar model based on extracted speech features, including machine learning algorithms.

[0118] "Means for generating images based on an avatar model" refers to technology for generating the final avatar image based on the selected avatar model, including deep learning models such as Generative Adversarial Networks (GANs) and StyleGANs.

[0119] "Means for checking and adjusting the quality of the generated avatar" refers to technology, including post-processing algorithms, for checking whether the generated avatar image is appropriate and correcting it if necessary.

[0120] "Means for transmitting generated avatar image data" refers to a communication means for transmitting the generated avatar image from the server to the user's device, such as the HTTP protocol.

[0121] "Means for displaying an avatar image" refers to technology for displaying the received avatar image on a user's device, including a display or monitor.

[0122] "Means for receiving user feedback" refers to an interface for collecting user opinions and comments about the generated avatar, including a GUI or web form.

[0123] "FFT analysis" refers to the fast Fourier transform technology used to efficiently analyze the frequency components of audio data.

[0124] "Means for generating avatar images using GAN" refers to technology for automatically generating natural-looking avatar images using a generative artificial network (GAN).

[0125] "Post-processing algorithms" refers to post-processing techniques that apply additional adjustments and modifications to the generated images, including quality improvement using OpenCV.

[0126] MODE FOR CARRYING OUT THE INVENTION

[0127] The present invention is a system that generates realistic and natural avatar images based on voice input. The series of processes that generate an avatar from a user's voice input and display that image on a terminal will be described below.

[0128] First, the user speaks into the device. This voice input is collected using a microphone and voice recognition software. For example, the user might give a voice command such as, "Hello, I'm Suzuki. Please create an anime character for me." This voice data is then digitally recorded on the device.

[0129] The device then sends the recorded audio data to the server using an HTTP POST request over the internet connection, in a common audio file format such as WAV.

[0130] The server performs preprocessing on the received audio data. This preprocessing includes noise reduction, which uses filters to remove background noise and unwanted sounds to clean the audio data. It also analyzes the frequency components of the audio data using FFT (Fast Fourier Transform). This allows it to extract audio features such as pitch (high and low pitch), formants (resonant frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0131] Based on the extracted features, the server selects the optimal avatar model. This selection is performed using a machine learning algorithm, such as a random forest or deep neural network, to match the voice features with the avatar model. If the user's voice has a youthful tone, a youthful anime character model will be selected.

[0132] After the model is selected, the server generates an avatar image using a generative AI model such as a Generative Adversarial Network (GAN) or StyleGAN. This process generates a natural-looking avatar that reflects the audio features. The generated avatar image is then quality checked and adjusted using post-processing algorithms. For example, libraries such as OpenCV are used to correct unnatural facial distortions and imperfections in facial expressions.

[0133] The final avatar image is sent from the server to the device. Again, the HTTP protocol is generally used. The device displays the received avatar image to the user. The user can check the displayed avatar and provide feedback if necessary. This feedback is reflected in the next avatar generation, enabling the generation of a higher quality avatar.

[0134] Specific examples

[0135] Use case 1: Anime-style avatar generation

[0136] Example prompt sentence:

[0137] "Hello, I'm Suzuki. Please make me an anime character."

[0138] The system allows users to quickly generate natural voice-based avatars to enhance visual communication, a process achieved through the use of highly accurate voice analysis and generative AI models.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Program processing flow

[0141] Step 1:

[0142] The user speaks to the device, for example, "Hello, my name is Sato. Please create an anime character for me." At this time, the voice input is recorded as digital voice data via a microphone.

[0143] Input: User's voice

[0144] Output: Digital audio data (WAV, etc.)

[0145] Step 2:

[0146] The device sends the recorded audio data in digital format to the server using an HTTP POST request.

[0147] Input: Digital audio data

[0148] Output: Audio data sent in HTTP POST request

[0149] Step 3:

[0150] The server first denoises the received audio data to convert it into an analyzable format, using a filter to remove background noise and obtain clean audio data, then resamples the audio data to a standard sampling rate.

[0151] Input: Transmitted audio data

[0152] Output: Preprocessed, clean audio data

[0153] Step 4:

[0154] The server analyzes the frequency components from the preprocessed audio data using FFT (Fast Fourier Transform), extracting audio features such as pitch, formants, tone, and rhythm. The features are stored in list or vector format.

[0155] Input: Preprocessed clean audio data

[0156] Output: A list or vector of audio features

[0157] Step 5:

[0158] The server selects the optimal avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest or deep neural network) and references a database of voice features and avatar models. The selected model is recorded as the selection result.

[0159] Input: List of audio features

[0160] Output: Selected avatar model

[0161] Step 6:

[0162] The server generates an avatar image based on the selected avatar model using a generative AI model (e.g., GAN or StyleGAN). The generation process uses multiple neural networks to generate the final avatar image.

[0163] Input: Selected avatar model

[0164] Output: Generated avatar image

[0165] Step 7:

[0166] The server performs a quality check on the generated avatar images, detecting any unnatural features (facial distortions or imperfections in expressions) and correcting them using post-processing algorithms (e.g., OpenCV).

[0167] Input: Generated avatar image

[0168] Output: Quality adjusted avatar image

[0169] Step 8:

[0170] The server then sends the final avatar image data to the user's device, again using the HTTP protocol.

[0171] Input: Quality adjusted avatar image

[0172] Output: Avatar image data sent in the HTTP response

[0173] Step 9:

[0174] The device displays the received avatar image to the user. The avatar image can be visually confirmed using the display. The user can check the displayed avatar image and provide feedback through the device.

[0175] Input: Avatar image data received in the HTTP response

[0176] Output: Avatar image displayed on the screen

[0177] Through this series of steps, users can quickly generate realistic and natural avatar images based on voice input, enhancing visual communication.

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] In conventional virtual stores, there has been no system that provides an assistant avatar that can interact with users in real time. This has led to the issue of not being able to improve the interactive shopping experience in stores. There has also been a lack of systems that can generate and display an appropriate avatar based on the user's voice request.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes a voice input means, a voice data transmission means, a voice data preprocessing means, a voice feature extraction means, a voice feature selection means, an avatar model selection means based on the voice feature, an image generation means based on the avatar model, a quality check and adjustment means for the generated avatar, a transmission means for the generated avatar image data, a display means for the avatar image, a user feedback receiving means, and a generation means for an assistant avatar to provide an interactive shopping experience in a virtual store. This allows a user to easily generate an avatar through voice input and realize an interactive shopping experience.

[0183] "Voice input means" refers to a device for capturing the user's voice, and includes a microphone and the voice input function of a smartphone.

[0184] "Means for transmitting audio data" refers to a communications interface or network protocol for transmitting recorded audio data to a server or other processing system.

[0185] "Audio data preprocessing means" refers to algorithms or software that remove noise from recorded audio data and convert the data into a format that is easier to analyze.

[0186] "Means for extracting speech features" refers to technology for extracting characteristic elements of speech, such as pitch, formants, tone, and rhythm, from speech data.

[0187] The "means for selecting an avatar model based on speech features" refers to a machine learning algorithm or database for selecting the optimal avatar model based on the extracted speech features.

[0188] "Means for generating images based on an avatar model" refers to technologies for generating actual avatar images from a selected avatar model, and includes deep learning technologies such as GAN (generative adversarial networks) and StyleGAN.

[0189] "Means for checking and adjusting the quality of the generated avatar" refers to algorithms or software for evaluating the quality of the generated avatar image and correcting any unnatural parts.

[0190] The "means for transmitting generated avatar image data" refers to a communication interface or network protocol for transmitting the avatar image data generated by the server to a user's device.

[0191] The "means for displaying an avatar image" refers to a display device or application that a user uses to view the generated avatar image.

[0192] "Means for receiving user feedback" refers to an interface or system for collecting evaluations and opinions from users.

[0193] "Means for generating assistant avatars to provide an interactive shopping experience within a virtual store" refers to a technology that generates an avatar based on voice data to interact with users in a virtual store environment and provide guidance and support.

[0194] A system for implementing the present invention includes a voice input means, a means for transmitting voice data, a means for preprocessing the voice data, a means for extracting voice features, a means for selecting an avatar model based on the voice features, a means for generating an image based on the avatar model, a means for checking and adjusting the quality of the generated avatar, a means for transmitting the generated avatar image data, a means for displaying the avatar image, and a means for receiving user feedback.

[0195] First, the user uses a voice input means such as a smartphone or a head-mounted display to input a voice such as "Hello, I'm looking for shoes." The voice data is recorded through the microphone of the device and transmitted to the server via the network by a means for transmitting voice data.

[0196] The server preprocesses the received audio data, removing noise and converting it to digital form. Specifically, it uses the Python speech_recognition library. From the preprocessed audio data, the server extracts audio features. Using FFT (Fast Fourier Transform) and cepstrum analysis, audio features such as pitch, formant, tone, and rhythm are extracted.

[0197] Based on the extracted voice features, the server selects the optimal avatar model using machine learning algorithms and pre-trained voice-character pair data. For example, if the voice tone is bright and high, a youthful anime character model will be selected at this stage.

[0198] Based on the selected avatar model, the server generates an avatar image using deep learning techniques (e.g., GAN or StyleGAN). The generated avatar image is then quality-evaluated by the server to ensure there are no unnatural features. If there are any defects, they are corrected using a post-processing algorithm.

[0199] The final avatar image data after adjustment is sent from the server to the device. The device displays the received avatar image on the user's display device. The user can review the avatar and provide feedback if necessary. This feedback is sent to the server and is used to improve the system and increase the accuracy of avatar generation.

[0200] As a specific example of use, consider a situation where a user speaks to a device, "Hello, I'm Suzuki. Please create an anime character for me." The server selects an avatar model from this voice data, generates an anime-style avatar using GAN, and finally displays it on the device as a series of processes.

[0201] Another example of a prompt when generating an avatar using a generative AI model is, "Please generate an avatar that is a youthful female character with a friendly smile."

[0202] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0203] Step 1:

[0204] The user inputs voice into the device. This voice input includes the characteristics and wishes of the avatar the user wants to create. For example, the user might say, "Hello, my name is Sato. Please create an anime character for me." The voice data is recorded via a smartphone or microphone. The input is voice data, and the output is a recorded voice file.

[0205] Step 2:

[0206] The device sends the recorded audio data to the server, using communication methods such as the Internet or Bluetooth. The input is an audio file, and the output is the transmission of audio data to the server.

[0207] Step 3:

[0208] The server preprocesses the received audio data. Specifically, it performs noise removal and digital conversion of the audio signal. This preprocessing uses the speech_recognition library and other audio processing algorithms. The input is audio data, and the output is the preprocessed audio data.

[0209] Step 4:

[0210] The server extracts features from the speech data. For example, it uses FFT (Fast Fourier Transform) and cepstrum analysis techniques to extract speech features such as pitch, formant, tone, and rhythm. The input is preprocessed speech data, and the output is speech features.

[0211] Step 5:

[0212] The server selects an avatar model based on the extracted voice features. Here, a machine learning algorithm is used to compare the extracted voice features with pre-trained data and select the optimal avatar model. The input is the voice features, and the output is the selected avatar model.

[0213] Step 6:

[0214] The server generates an image based on the selected avatar model. Specifically, it uses a generative AI model such as GAN or StyleGAN to generate the avatar image. It uses a prompt such as "Please generate an avatar that is a youthful female character with a friendly smile." The input is the avatar model, and the output is the generated avatar image data.

[0215] Step 7:

[0216] The server checks the quality of the generated avatar and adjusts any unnatural parts. Quality assessment algorithms and post-processing algorithms are used here. The input is the generated avatar image data, and the output is the adjusted avatar image data.

[0217] Step 8:

[0218] The server sends the adjusted avatar image data to the user's device. Data is transferred using a network protocol. The input is the adjusted avatar image data, and the output is data sent to the device.

[0219] Step 9:

[0220] The device displays the received avatar image. Specifically, it is displayed on a smartphone display or a head-mounted display. The input is avatar image data, and the output is the displayed avatar image.

[0221] Step 10:

[0222] The user checks the generated avatar image and sends any feedback they wish to provide to the server via their device. The input is the user's feedback, and the output is the feedback data sent to the server.

[0223] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0224] The present invention aims to generate natural and realistic avatar images that reflect the user's emotions by combining an emotion engine with a system that generates avatars from voice data. This system is implemented using the following components and procedures.

[0225] System Overview

[0226] The system includes the following major components:

[0227] Voice input means

[0228] A means of transmitting audio data

[0229] A means of preprocessing audio data

[0230] A means of extracting speech features

[0231] Emotion Engine

[0232] A method for selecting avatar models based on speech features and emotions

[0233] Avatar model and means for generating images based on emotions

[0234] A way to check and adjust the quality of the generated avatars

[0235] A means for transmitting the generated avatar image data

[0236] A means of displaying an avatar image

[0237] A way to receive user feedback

[0238] Program processing

[0239] The program processing in this system will be explained in natural language below.

[0240] 1. Audio input

[0241] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Sato. Please create a lively anime character."

[0242] 2. Sending audio data

[0243] The device records the user's voice and generates digital audio data, which is saved in a common audio file format (e.g., WAV, MP3) and sent to a server.

[0244] 3. Preprocessing of audio data

[0245] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[0246] The audio data is then converted into a digital format that makes it easier to analyze.

[0247] 4. Extraction of speech features

[0248] The server extracts features from the preprocessed speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0249] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[0250] 5. Emotional Recognition

[0251] The emotion engine recognizes the user's emotions from the voice data. The emotion engine analyzes voice parameters and distinguishes emotions such as joy, sadness, anger, and surprise.

[0252] Machine learning algorithms (e.g., neural networks, RNNs) are used to identify emotions from a pre-trained dataset.

[0253] 6. Select an avatar model

[0254] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character.

[0255] To select a model, a machine learning algorithm is used to select the optimal model from a pre-trained database of voice-character pairs.

[0256] 7. Image Generation

[0257] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[0258] 8. Avatar quality check and adjustment

[0259] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[0260] 9. Sending and displaying avatar images

[0261] The server generates the final avatar image data and sends it to the user's device, where it is saved in high-resolution format and provided to the user.

[0262] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[0263] Specific examples

[0264] Use case 1: Generating anime-style avatars that reflect emotions

[0265] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[0266] 2. The device records the audio and sends it to the server.

[0267] 3. The server preprocesses the audio data, removes noise, and extracts features.

[0268] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[0269] 5. The server selects a lively anime character model based on the features and emotions.

[0270] 6. The server uses GAN to generate an animated avatar with a happy expression.

[0271] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[0272] 8. The server sends the avatar image to the device.

[0273] 9. The device displays the avatar to the user, who confirms it.

[0274] The system allows users to quickly generate natural, high-quality, and emotionally sensitive avatars based on their voice.

[0275] The processing flow will be explained below.

[0276] Step 1:

[0277] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Suzuki. Please create a lively anime character."

[0278] Step 2:

[0279] The device records the user's voice and generates digital audio data that is stored in common audio file formats such as WAV or MP3.

[0280] Step 3:

[0281] The device sends the recorded audio data to the server, using an internet connection to transfer the data in a reliable manner (e.g. HTTPS).

[0282] Step 4:

[0283] The server receives the audio data and first performs noise reduction, using digital signal processing techniques (e.g. bandpass filters) to filter out background noise and unwanted sounds.

[0284] Step 5:

[0285] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are analyzed using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[0286] Step 6:

[0287] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes voice parameters to identify emotions such as joy, sadness, anger, and surprise. Specifically, it uses a machine learning algorithm (e.g., neural network, RNN) to identify emotions from a pre-trained dataset.

[0288] Step 7:

[0289] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character. The model selection uses a machine learning algorithm and references a pre-trained voice-character pair database.

[0290] Step 8:

[0291] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[0292] Step 9:

[0293] The server checks the quality of the generated avatar, specifically checking for facial distortions and unnatural expressions, and automatically corrects any unnatural features using post-processing algorithms.

[0294] Step 10:

[0295] The server sends the final avatar image data to the device, where it is stored in high-resolution format and may be compressed depending on the requirements of the service being used.

[0296] Step 11:

[0297] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything they don't like about it. For example, the user can make a specific request, such as "I want the facial expression to be more smiling."

[0298] Specific examples

[0299] Use case 1: Generating anime-style avatars that reflect emotions

[0300] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[0301] 2. The device records the audio and sends it to the server.

[0302] 3. The server preprocesses the audio data, removes noise, and extracts features.

[0303] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[0304] 5. The server selects a lively anime character model based on the features and emotions.

[0305] 6. The server uses GAN to generate an animated avatar with a happy expression.

[0306] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[0307] 8. The server sends the avatar image to the device.

[0308] 9. The device displays the avatar to the user, who confirms it.

[0309] Example 2

[0310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] Conventional avatar generation systems have had difficulty generating natural and realistic avatar images that reflect the user's emotions. A particular technical challenge has been properly recognizing emotions from voice data and creating an avatar with corresponding facial expressions. Therefore, a system that can easily generate avatars that accurately reflect the user's intentions and emotions has been sought.

[0312] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0313] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, means for recognizing emotions using an emotion engine, means for selecting an avatar model based on the voice features and emotions, means for generating an image based on the avatar model and emotions, means for checking and adjusting the quality of the generated avatar, and means for transmitting the generated avatar image data. This makes it possible to recognize a user's emotions from voice data and quickly generate natural and realistic avatar images that reflect those emotions.

[0314] The "voice input means" is a means that enables a user to utter voice data into the microphone of the terminal.

[0315] The "means for transmitting audio data" refers to a means for transferring recorded audio data to a server in digital format.

[0316] "Means for preprocessing audio data" refers to means for removing noise and filtering audio data, and converting the audio data into a format that is easy to analyze.

[0317] The "means for extracting speech features" refers to a means for extracting features such as pitch, formant, tone, and rhythm from speech data.

[0318] The "emotion engine" is an engine that uses machine learning algorithms to analyze the user's emotions from voice data and identify emotions such as joy, sadness, anger, and surprise.

[0319] The "means for selecting an avatar model" is a means for selecting an optimal avatar model based on speech features and emotions.

[0320] The "means for generating an image" is a means using a deep learning model to generate an avatar image based on a selected avatar model and a recognized emotion.

[0321] The "means for checking and adjusting the quality of the generated avatar" refers to a means for checking the quality of the generated avatar image and adjusting facial distortions and unnatural expressions using a post-processing algorithm.

[0322] The "means for transmitting generated avatar image data" refers to a means for transmitting the final avatar image data to a user's device.

[0323] The "means for displaying an avatar image" refers to a means for displaying the received avatar image on the user's terminal.

[0324] The "means for receiving user feedback" is a means for a user to provide opinions or requests for corrections to the generated avatar image.

[0325] The present invention is a system that recognizes a user's emotions from voice data and generates a natural and realistic avatar image that reflects those emotions. This system is composed of multiple hardware and software components. Specifically, it is implemented using the following components and procedures.

[0326] Main components

[0327] Voice input method: The user speaks into the device's microphone, describing the desired avatar's characteristics, messages, and emotions. The device captures the voice using a microphone sound input library (e.g., PyAudio).

[0328] How audio data is transmitted: The device saves the recorded audio data in a common audio file format (e.g., WAV, MP3) and transmits it to the server using an HTTP request.

[0329] A means of pre-processing the audio data: The server uses an audio processing library (e.g., Librosa) to denoise and filter the audio data, then convert it into a digital format that is easier to analyze.

[0330] Method for extracting speech features: The server uses FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formant, tone, and rhythm from the speech data.

[0331] Emotion engine: The server uses an emotion engine (e.g., neural network or RNN) to analyze the user's emotions from the voice features and identify emotions such as joy, sadness, anger, and surprise.

[0332] Method for selecting an avatar model: The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on audio features and emotion information.

[0333] Means for generating images: The server uses GAN (generative adversarial network) or StyleGAN to generate avatar images based on the selected avatar model and recognized emotions.

[0334] A means to check and adjust the quality of the generated avatar: The server checks the generated avatar image for facial distortion and unnatural expressions, and performs post-processing using image processing algorithms (e.g., OpenCV) to make any necessary fine-tuning.

[0335] Means for sending the generated avatar image data: The server saves the final avatar image data in a high-resolution format (e.g., PNG) and sends it to the user's device.

[0336] A method for displaying the avatar image: The device displays the received avatar image. For example, HTML5 Use tags to display in the browser.

[0337] A means of receiving user feedback: Users can provide specific opinions and requests for corrections to the displayed avatar image. For example, they can send feedback via their device, such as "I want the facial expression to be more smiling."

[0338] Usage example

[0339] Specific examples are shown below.

[0340] Use case 1: Generating anime-style avatars that reflect emotions

[0341] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[0342] 2. The device records the audio and sends it to the server in WAV format.

[0343] 3. The server uses Librosa to denoise and filter the audio data, making it easier to analyze.

[0344] 4. The server extracts features from the audio data using FFT and cepstrum analysis.

[0345] 5. The emotion engine identifies the user's emotion "joy" from the voice.

[0346] 6. The server uses random forests to select a model of a lively anime character based on features and emotions.

[0347] 7. The server uses GAN to generate an animated avatar with a happy expression.

[0348] 8. The server checks the quality of the generated avatar using OpenCV and makes adjustments if necessary.

[0349] 9. The server sends the final avatar image to the device.

[0350] 10. The device displays the received image and the user confirms it.

[0351] Prompt Sentence Examples

[0352] Generate a new avatar image using the voice data of a user saying "Hello, I'm Suzuki. Please create a lively anime character." The emotion to use is "joy."

[0353] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0354] Step 1:

[0355] The user speaks into the device's microphone, including the desired avatar's characteristics, message, and emotion. For example, "Hello, I'm Sato. Please create a lively anime character." This voice is input as an analog signal.

[0356] Step 2:

[0357] The device uses a microphone audio input library (e.g., PyAudio) to capture the user's voice and save it in a digital format (WAV or MP3). The saved voice data is then output.

[0358] Step 3:

[0359] The device sends audio data to the server via an HTTP request (e.g., using Python's requests library). This operation inputs digital audio data and transfers it to the server.

[0360] Step 4:

[0361] The server denoises the audio data using an audio processing library such as Librosa, filtering out background noise and unwanted sounds. The input is digital audio data, and the output is clean audio data after noise reduction.

[0362] Step 5:

[0363] The server uses an audio processing library to convert the clean audio data into a more analyzable form. This calculation includes formatting in digital form. The converted audio data is output.

[0364] Step 6:

[0365] The server performs FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formants, tone, and rhythm. The input is preprocessed speech data, and the extracted speech features are output.

[0366] Step 7:

[0367] The emotion engine analyzes the user's emotions using a machine learning algorithm (e.g., neural network, RNN) based on the extracted voice features. The input is the voice features, and the output is the identified emotion (e.g., joy, sadness).

[0368] Step 8:

[0369] The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on the identified emotions and voice features. The input is the voice features and emotion information, and the selected avatar model is output.

[0370] Step 9:

[0371] The server generates an avatar image using a deep learning model (e.g., GAN, StyleGAN) based on the selected avatar model and the identified emotion. The input is the avatar model and emotion information, and the generated avatar image is output.

[0372] Step 10:

[0373] The server uses image processing algorithms (e.g., OpenCV) to check the quality of the generated avatar image and check for facial distortions and unnatural expressions. If necessary, post-processing is performed and a fine-tuned avatar image is output.

[0374] Step 11:

[0375] The server saves the final avatar image in a high-resolution format (e.g., PNG) and sends it to the user's device as an HTTP response. The input is the adjusted avatar image data, and the output is the sent image data.

[0376] Step 12:

[0377] The device receives an avatar image and sends it to HTML5. It displays the image to the user using tags, etc. The user can review the image and provide specific feedback (e.g., "Please smile more") if there is anything they don't like. The input is avatar image data, and the displayed avatar image is output.

[0378] (Application example 2)

[0379] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0380] Conventional avatar generation systems have had difficulty generating avatars that accurately reflect a user's emotions in real time. In particular, live streaming and real-time content distribution require immediate reflection of the user's emotions and voice characteristics. To solve this problem, a system is needed that performs highly accurate emotion analysis and voice feature extraction, selects an avatar model based on the results, and generates images in real time.

[0381] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0382] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, and means for analyzing emotions, which enable real-time analysis of emotions from a user's voice, selection of an avatar model based on the emotions, and generation of an image, thereby enabling the generation of avatars that accurately reflect the user's emotions in live streaming or real-time content distribution.

[0383] The "voice input means" is a device or mechanism for acquiring the user's voice as digital voice data.

[0384] The "means for transmitting voice data" is a mechanism for transmitting the acquired voice data to another device or server.

[0385] "Means for preprocessing audio data" refers to a processing device or method for removing noise from audio data and making it easier to analyze.

[0386] A "means for extracting audio features" is a device or method for extracting characteristic data such as formants, pitch, tone, and rhythm from audio data.

[0387] "Means for analyzing emotions" refers to devices or methods for identifying and determining a user's emotions from voice data.

[0388] The "means for selecting an avatar model based on speech features and emotions" refers to a device or method for selecting an optimal avatar model based on the acquired speech features and analyzed emotions.

[0389] The "means for generating an image based on an avatar model and an emotion" refers to a device or method for generating a specific avatar image according to a selected avatar model and an analyzed emotion.

[0390] "Means for checking and adjusting the quality of the generated avatar" refers to a device or method for inspecting the generated avatar image for unnatural features or defects and correcting them if necessary.

[0391] The "means for transmitting the generated avatar image data" is a mechanism for transmitting the generated avatar image data to a user's terminal or another system.

[0392] The "means for displaying an avatar image" refers to a display device or a display program for visually displaying the generated avatar image.

[0393] The "means for receiving user feedback" refers to a device or interface for obtaining user opinions and impressions regarding the generated avatar image.

[0394] This invention relates to an avatar generation system that combines voice data and emotion analysis. The purpose of this system is to generate and display avatars in real time that reflect the user's emotions in real-time live streaming and content distribution.

[0395] Main system components

[0396] The system requires the following major components:

[0397] Voice input means

[0398] A means of transmitting audio data

[0399] A means of preprocessing audio data

[0400] A means of extracting speech features

[0401] A means of analyzing emotions

[0402] A method for selecting avatar models based on speech features and emotions

[0403] Avatar model and means for generating images based on emotions

[0404] A way to check and adjust the quality of the generated avatars

[0405] A means for transmitting the generated avatar image data

[0406] A means of displaying an avatar image

[0407] A way to receive user feedback

[0408] Hardware and software used

[0409] This system requires high-performance hardware and advanced software. The main hardware and software used are described below.

[0410] Hardware

[0411] Smartphone: An iOS or Android device used to input voice and display results.

[0412] High-performance server: Equipped with a GPU, it performs data preprocessing, voice analysis, emotion analysis, and avatar generation.

[0413] software

[0414] Audio processing libraries: e.g. librosa, used for preprocessing audio data (noising, filtering).

[0415] Machine learning frameworks: e.g., TensorFlow, PyTorch, used for audio feature extraction and sentiment analysis.

[0416] Deep learning models: e.g. GAN (generative artificial network), StyleGAN, used to generate avatar images.

[0417] Speech recognition models, such as DeepSpeech, are used to extract features from speech data.

[0418] Data processing flow

[0419] The server first receives the voice data and performs preprocessing. Next, it extracts voice features and analyzes emotions. Based on this information, it selects an avatar model and generates an avatar image in real time. The generated image undergoes a quality check and is then sent to the user's smartphone. The smartphone displays the avatar image and receives user feedback.

[0420] Specific examples

[0421] Example 1: Real-time avatar generation during live streaming

[0422] When a user starts live streaming on their smartphone, they begin speaking into the voice input means. For example, they might say, "Hello, I'm so happy today!" This speech is sent to the server in real time. The server preprocesses the speech data and extracts features to analyze emotions. Based on the analysis results, an avatar model is selected and an avatar image is generated in real time. This avatar image is sent to the smartphone and visually displayed to the user.

[0423] Prompt Sentence Examples

[0424] "Hello, I'm so happy today!"

[0425] => Emotion: Joy

[0426] => Avatar Model: Smiling Anime Character

[0427] This detailed processing procedure generates a high-quality avatar that reflects the user's emotions in real time, enabling viewers to experience immersive live streaming.

[0428] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0429] Step 1:

[0430] The user begins speaking into the smartphone's microphone. The input is the user's voice data, for example, "Hello, I'm very happy today!" The output is recorded digital voice data. The device transmits this voice data to the server in real time.

[0431] Step 2:

[0432] The server receives the audio data sent from the terminal. The input is the digital audio data sent from the terminal. The server first preprocesses this audio data and performs noise removal. As a result, clean audio data with noise removed is obtained as output.

[0433] Step 3:

[0434] The server extracts features from the preprocessed speech data. The input is clean speech data with noise removed. Specifically, it uses FFT (Fast Fourier Transform) and cepstrum analysis to extract characteristic data such as formants, pitch, tone, and rhythm. The output is these speech feature data.

[0435] Step 4:

[0436] The server performs emotion analysis based on the speech feature data. The input is speech feature data. Neural networks or RNNs are used for emotion analysis to analyze emotions such as joy, sadness, anger, and surprise. The output is the analyzed emotion data.

[0437] Step 5:

[0438] The server selects the optimal avatar model based on the extracted speech features and analyzed emotion data. The input is speech feature data and emotion data. The server uses a machine learning algorithm to select the optimal avatar model from a pre-trained database. The output is the selected avatar model.

[0439] Step 6:

[0440] The server generates an avatar image based on the selected avatar model and the analyzed emotion data. The input is the avatar model and emotion data. The server uses a deep learning model (e.g., GAN, StyleGAN) to generate the avatar's face and expressions. The output is the generated avatar image.

[0441] Step 7:

[0442] The server checks the quality of the generated avatar image. The input is the generated avatar image. The server checks for facial distortion and unnatural expressions, and makes adjustments using post-processing algorithms if necessary. The output is the final quality-assured avatar image.

[0443] Step 8:

[0444] The server sends the final avatar image data to the user's smartphone. The input is a quality-guaranteed avatar image. The output is the avatar image displayed on the smartphone.

[0445] Step 9:

[0446] The device visually displays the received avatar image to the user. The input is the avatar image data sent from the server. The user can review the avatar image and provide feedback. The feedback is sent to the system through a robust user interface and used for further improvement.

[0447] Specific prompt examples:

[0448] "Hello, I'm so happy today!"

[0449] => Emotion: Joy

[0450] => Avatar Model: Smiling Anime Character

[0451] This processing step generates high-quality avatar images that reflect the user's emotions in real time, providing a more lifelike experience for the user and viewers.

[0452] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0453] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0454] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0455] [Second embodiment]

[0456] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0457] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0458] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0459] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0460] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0461] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0462] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0463] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0464] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0465] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0466] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0467] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0468] The present invention provides a system for generating avatars from voice data. The system aims to generate natural and realistic avatar images from a user's voice input, and is implemented by the following method and procedure.

[0469] System Overview

[0470] The system includes the following major components:

[0471] Voice input means

[0472] A means of transmitting audio data

[0473] A means of preprocessing audio data

[0474] A means of extracting speech features

[0475] A method for selecting avatar models based on speech features

[0476] A method for generating images based on avatar models

[0477] A way to check and adjust the quality of the generated avatars

[0478] A means for transmitting the generated avatar image data

[0479] A means of displaying an avatar image

[0480] A way to receive user feedback

[0481] Program processing

[0482] The program processing in this system will be explained in natural language below.

[0483] 1. Audio input

[0484] The user speaks into the device, and can include the message they want to convey and the characteristics of the avatar they want to create. For example, they might say, "Hello, my name is Sato. Please create an anime character for me."

[0485] 2. Sending audio data

[0486] The terminal records the user's voice and transmits the digital voice data to the server.

[0487] 3. Preprocessing of audio data

[0488] The server receives the audio data and first performs noise reduction, which filters out background noise and unwanted sounds.

[0489] The audio data is then converted into a digital format that makes it easier to analyze.

[0490] 4. Extraction of speech features

[0491] The server extracts speech features from the speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0492] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[0493] 5. Select an avatar model

[0494] The server selects the most suitable avatar model based on the extracted voice features. For example, if the voice is bright and high-pitched, it selects a youthful anime character.

[0495] To select a model, a machine learning algorithm is used, referencing pre-trained voice-character pair data.

[0496] 6. Image Generation

[0497] The server generates an image based on the selected avatar model, specifically using deep learning models such as GAN (Generative Adversarial Network) and StyleGAN.

[0498] This generates a facial pattern corresponding to the audio features.

[0499] 7. Avatar quality check and adjustment

[0500] The server checks the quality of the generated avatar, checking for unnatural features such as facial distortion or unnatural facial expressions.

[0501] If necessary, fine-tune using post-processing algorithms.

[0502] 8. Sending and displaying avatar images

[0503] The server generates the final avatar image data and sends it to the user's terminal.

[0504] The device displays the received avatar image to the user, who can then review the avatar image and provide feedback if necessary.

[0505] Specific examples

[0506] Below are some specific usage examples.

[0507] Use case 1: Anime-style avatar generation

[0508] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create an anime character for me."

[0509] 2. The device records the audio and sends it to the server.

[0510] 3. The server preprocesses the audio data, removes noise, and extracts features.

[0511] 4. The server selects a youthful anime character model based on the features.

[0512] 5. The server uses GAN to generate an animated avatar.

[0513] 6. The server checks the quality of the generated avatar and makes adjustments if necessary.

[0514] 7. The server sends the avatar image to the device.

[0515] 8. The device displays the avatar to the user, who confirms it.

[0516] This system allows users to quickly generate a natural voice-based avatar to enhance visual communication.

[0517] The processing flow will be explained below.

[0518] Step 1:

[0519] The user speaks into the device's microphone, describing the desired avatar's characteristics and a message. For example, they might say, "Hello, I'm Sato. Please create an anime character for me."

[0520] Step 2:

[0521] The device records the user's voice and generates digital audio data that is stored in a common audio file format (e.g., WAV, MP3).

[0522] Step 3:

[0523] The device sends the recorded audio data to a server, using an internet connection for data transfer.

[0524] Step 4:

[0525] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[0526] Step 5:

[0527] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are extracted using FFT (Fast Fourier Transform) and cepstrum analysis.

[0528] Step 6:

[0529] The server selects an appropriate avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest, SVM) to select the best model from a pre-trained voice-character pair database.

[0530] Step 7:

[0531] The server generates an avatar image based on the selected avatar model, and uses a deep learning model (e.g., GAN, StyleGAN) to generate a facial pattern corresponding to the audio features.

[0532] Step 8:

[0533] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[0534] Step 9:

[0535] The server sends the final avatar image data to the device, where it is saved in high-resolution format and provided to the user.

[0536] Step 10:

[0537] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[0538] This series of steps allows users to quickly generate a natural, high-quality avatar based on their voice.

[0539] Example 1

[0540] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0541] Conventional avatar generation systems have had many problems when generating natural and realistic avatars from user voice, particularly in the preprocessing of voice data, feature extraction, and final image generation and quality adjustment. Specifically, the efficiency of noise removal from voice data was low, and the accuracy of extracted features was insufficient, making it often impossible to select the optimal avatar model. Furthermore, the quality of the generated images was unstable, with many unnatural parts. A method to resolve these issues is needed.

[0542] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0543] In this invention, the server includes means for performing FFT analysis on the voice data, means for generating avatar images using GAN, and means for adjusting the quality of the generated images using a post-processing algorithm, which enables highly accurate analysis and noise removal of the voice data, selection of the optimal avatar model, and generation and adjustment of high-quality avatar images.

[0544] "Voice input means" refers to a device or interface that allows a user to input voice, such as a microphone or voice recognition software.

[0545] "Means for transmitting audio data" refers to the communications means for transmitting audio data from the user's device to the server, typically including an internet connection and the HTTP protocol.

[0546] "Means for pre-processing audio data" refers to processes for converting recorded audio data into an analyzable format, including, for example, noise reduction and digital conversion.

[0547] "Means for extracting speech features" refers to techniques for analyzing and extracting necessary speech features from preprocessed speech data, including FFT (Fast Fourier Transform) and cepstrum analysis.

[0548] "Means for selecting an avatar model based on speech features" refers to an algorithm for selecting an optimal avatar model based on extracted speech features, including machine learning algorithms.

[0549] "Means for generating images based on an avatar model" refers to technology for generating the final avatar image based on the selected avatar model, including deep learning models such as Generative Adversarial Networks (GANs) and StyleGANs.

[0550] "Means for checking and adjusting the quality of the generated avatar" refers to technology, including post-processing algorithms, for checking whether the generated avatar image is appropriate and correcting it if necessary.

[0551] "Means for transmitting generated avatar image data" refers to a communication means for transmitting the generated avatar image from the server to the user's device, such as the HTTP protocol.

[0552] "Means for displaying an avatar image" refers to technology for displaying the received avatar image on a user's device, including a display or monitor.

[0553] "Means for receiving user feedback" refers to an interface for collecting user opinions and comments about the generated avatar, including a GUI or web form.

[0554] "FFT analysis" refers to the fast Fourier transform technology used to efficiently analyze the frequency components of audio data.

[0555] "Means for generating avatar images using GAN" refers to technology for automatically generating natural-looking avatar images using a generative artificial network (GAN).

[0556] "Post-processing algorithms" refers to post-processing techniques that apply additional adjustments and modifications to the generated images, including quality improvement using OpenCV.

[0557] MODE FOR CARRYING OUT THE INVENTION

[0558] The present invention is a system that generates realistic and natural avatar images based on voice input. The series of processes that generate an avatar from a user's voice input and display that image on a terminal will be described below.

[0559] First, the user speaks into the device. This voice input is collected using a microphone and voice recognition software. For example, the user might give a voice command such as, "Hello, I'm Suzuki. Please create an anime character for me." This voice data is then digitally recorded on the device.

[0560] The device then sends the recorded audio data to the server using an HTTP POST request over the internet connection, in a common audio file format such as WAV.

[0561] The server performs preprocessing on the received audio data. This preprocessing includes noise reduction, which uses filters to remove background noise and unwanted sounds to clean the audio data. It also analyzes the frequency components of the audio data using FFT (Fast Fourier Transform). This allows it to extract audio features such as pitch (high and low pitch), formants (resonant frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0562] Based on the extracted features, the server selects the optimal avatar model. This selection is performed using a machine learning algorithm, such as a random forest or deep neural network, to match the voice features with the avatar model. If the user's voice has a youthful tone, a youthful anime character model will be selected.

[0563] After the model is selected, the server generates an avatar image using a generative AI model such as a Generative Adversarial Network (GAN) or StyleGAN. This process generates a natural-looking avatar that reflects the audio features. The generated avatar image is then quality checked and adjusted using post-processing algorithms. For example, libraries such as OpenCV are used to correct unnatural facial distortions and imperfections in facial expressions.

[0564] The final avatar image is sent from the server to the device. Again, the HTTP protocol is generally used. The device displays the received avatar image to the user. The user can check the displayed avatar and provide feedback if necessary. This feedback is reflected in the next avatar generation, enabling the generation of a higher quality avatar.

[0565] Specific examples

[0566] Use case 1: Anime-style avatar generation

[0567] Example prompt sentence:

[0568] "Hello, I'm Suzuki. Please make me an anime character."

[0569] The system allows users to quickly generate natural voice-based avatars to enhance visual communication, a process achieved through the use of highly accurate voice analysis and generative AI models.

[0570] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0571] Program processing flow

[0572] Step 1:

[0573] The user speaks to the device, for example, "Hello, my name is Sato. Please create an anime character for me." At this time, the voice input is recorded as digital voice data via a microphone.

[0574] Input: User's voice

[0575] Output: Digital audio data (WAV, etc.)

[0576] Step 2:

[0577] The device sends the recorded audio data in digital format to the server using an HTTP POST request.

[0578] Input: Digital audio data

[0579] Output: Audio data sent in HTTP POST request

[0580] Step 3:

[0581] The server first denoises the received audio data to convert it into an analyzable format, using a filter to remove background noise and obtain clean audio data, then resamples the audio data to a standard sampling rate.

[0582] Input: Transmitted audio data

[0583] Output: Preprocessed, clean audio data

[0584] Step 4:

[0585] The server analyzes the frequency components from the preprocessed audio data using FFT (Fast Fourier Transform), extracting audio features such as pitch, formants, tone, and rhythm. The features are stored in list or vector format.

[0586] Input: Preprocessed clean audio data

[0587] Output: A list or vector of audio features

[0588] Step 5:

[0589] The server selects the optimal avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest or deep neural network) and references a database of voice features and avatar models. The selected model is recorded as the selection result.

[0590] Input: List of audio features

[0591] Output: Selected avatar model

[0592] Step 6:

[0593] The server generates an avatar image based on the selected avatar model using a generative AI model (e.g., GAN or StyleGAN). The generation process uses multiple neural networks to generate the final avatar image.

[0594] Input: Selected avatar model

[0595] Output: Generated avatar image

[0596] Step 7:

[0597] The server performs a quality check on the generated avatar images, detecting any unnatural features (facial distortions or imperfections in expressions) and correcting them using post-processing algorithms (e.g., OpenCV).

[0598] Input: Generated avatar image

[0599] Output: Quality adjusted avatar image

[0600] Step 8:

[0601] The server then sends the final avatar image data to the user's device, again using the HTTP protocol.

[0602] Input: Quality adjusted avatar image

[0603] Output: Avatar image data sent in the HTTP response

[0604] Step 9:

[0605] The device displays the received avatar image to the user. The avatar image can be visually confirmed using the display. The user can check the displayed avatar image and provide feedback through the device.

[0606] Input: Avatar image data received in the HTTP response

[0607] Output: Avatar image displayed on the screen

[0608] Through this series of steps, users can quickly generate realistic and natural avatar images based on voice input, enhancing visual communication.

[0609] (Application example 1)

[0610] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0611] In conventional virtual stores, there has been no system that provides an assistant avatar that can interact with users in real time. This has led to the issue of not being able to improve the interactive shopping experience in stores. There has also been a lack of systems that can generate and display an appropriate avatar based on the user's voice request.

[0612] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0613] In this invention, the server includes a voice input means, a voice data transmission means, a voice data preprocessing means, a voice feature extraction means, a voice feature selection means, an avatar model selection means based on the voice feature, an image generation means based on the avatar model, a quality check and adjustment means for the generated avatar, a transmission means for the generated avatar image data, a display means for the avatar image, a user feedback receiving means, and a generation means for an assistant avatar to provide an interactive shopping experience in a virtual store. This allows a user to easily generate an avatar through voice input and realize an interactive shopping experience.

[0614] "Voice input means" refers to a device for capturing the user's voice, and includes a microphone and the voice input function of a smartphone.

[0615] "Means for transmitting audio data" refers to a communications interface or network protocol for transmitting recorded audio data to a server or other processing system.

[0616] "Audio data preprocessing means" refers to algorithms or software that remove noise from recorded audio data and convert the data into a format that is easier to analyze.

[0617] "Means for extracting speech features" refers to technology for extracting characteristic elements of speech, such as pitch, formants, tone, and rhythm, from speech data.

[0618] The "means for selecting an avatar model based on speech features" refers to a machine learning algorithm or database for selecting the optimal avatar model based on the extracted speech features.

[0619] "Means for generating images based on an avatar model" refers to technologies for generating actual avatar images from a selected avatar model, and includes deep learning technologies such as GAN (generative adversarial networks) and StyleGAN.

[0620] "Means for checking and adjusting the quality of the generated avatar" refers to algorithms or software for evaluating the quality of the generated avatar image and correcting any unnatural parts.

[0621] The "means for transmitting generated avatar image data" refers to a communication interface or network protocol for transmitting the avatar image data generated by the server to a user's device.

[0622] The "means for displaying an avatar image" refers to a display device or application that a user uses to view the generated avatar image.

[0623] "Means for receiving user feedback" refers to an interface or system for collecting evaluations and opinions from users.

[0624] "Means for generating assistant avatars to provide an interactive shopping experience within a virtual store" refers to a technology that generates an avatar based on voice data to interact with users in a virtual store environment and provide guidance and support.

[0625] A system for implementing the present invention includes a voice input means, a means for transmitting voice data, a means for preprocessing the voice data, a means for extracting voice features, a means for selecting an avatar model based on the voice features, a means for generating an image based on the avatar model, a means for checking and adjusting the quality of the generated avatar, a means for transmitting the generated avatar image data, a means for displaying the avatar image, and a means for receiving user feedback.

[0626] First, the user uses a voice input means such as a smartphone or a head-mounted display to input a voice such as "Hello, I'm looking for shoes." The voice data is recorded through the microphone of the device and transmitted to the server via the network by a means for transmitting voice data.

[0627] The server preprocesses the received audio data, removing noise and converting it to digital form. Specifically, it uses the Python speech_recognition library. From the preprocessed audio data, the server extracts audio features. Using FFT (Fast Fourier Transform) and cepstrum analysis, audio features such as pitch, formant, tone, and rhythm are extracted.

[0628] Based on the extracted voice features, the server selects the optimal avatar model using machine learning algorithms and pre-trained voice-character pair data. For example, if the voice tone is bright and high, a youthful anime character model will be selected at this stage.

[0629] Based on the selected avatar model, the server generates an avatar image using deep learning techniques (e.g., GAN or StyleGAN). The generated avatar image is then quality-evaluated by the server to ensure there are no unnatural features. If there are any defects, they are corrected using a post-processing algorithm.

[0630] The final avatar image data after adjustment is sent from the server to the device. The device displays the received avatar image on the user's display device. The user can review the avatar and provide feedback if necessary. This feedback is sent to the server and is used to improve the system and increase the accuracy of avatar generation.

[0631] As a specific example of use, consider a situation where a user speaks to a device, "Hello, I'm Suzuki. Please create an anime character for me." The server selects an avatar model from this voice data, generates an anime-style avatar using GAN, and finally displays it on the device as a series of processes.

[0632] Another example of a prompt when generating an avatar using a generative AI model is, "Please generate an avatar that is a youthful female character with a friendly smile."

[0633] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0634] Step 1:

[0635] The user inputs voice into the device. This voice input includes the characteristics and wishes of the avatar the user wants to create. For example, the user might say, "Hello, my name is Sato. Please create an anime character for me." The voice data is recorded via a smartphone or microphone. The input is voice data, and the output is a recorded voice file.

[0636] Step 2:

[0637] The device sends the recorded audio data to the server, using communication methods such as the Internet or Bluetooth. The input is an audio file, and the output is the transmission of audio data to the server.

[0638] Step 3:

[0639] The server preprocesses the received audio data. Specifically, it performs noise removal and digital conversion of the audio signal. This preprocessing uses the speech_recognition library and other audio processing algorithms. The input is audio data, and the output is the preprocessed audio data.

[0640] Step 4:

[0641] The server extracts features from the speech data. For example, it uses FFT (Fast Fourier Transform) and cepstrum analysis techniques to extract speech features such as pitch, formant, tone, and rhythm. The input is preprocessed speech data, and the output is speech features.

[0642] Step 5:

[0643] The server selects an avatar model based on the extracted voice features. Here, a machine learning algorithm is used to compare the extracted voice features with pre-trained data and select the optimal avatar model. The input is the voice features, and the output is the selected avatar model.

[0644] Step 6:

[0645] The server generates an image based on the selected avatar model. Specifically, it uses a generative AI model such as GAN or StyleGAN to generate the avatar image. It uses a prompt such as "Please generate an avatar that is a youthful female character with a friendly smile." The input is the avatar model, and the output is the generated avatar image data.

[0646] Step 7:

[0647] The server checks the quality of the generated avatar and adjusts any unnatural parts. Quality assessment algorithms and post-processing algorithms are used here. The input is the generated avatar image data, and the output is the adjusted avatar image data.

[0648] Step 8:

[0649] The server sends the adjusted avatar image data to the user's device. Data is transferred using a network protocol. The input is the adjusted avatar image data, and the output is data sent to the device.

[0650] Step 9:

[0651] The device displays the received avatar image. Specifically, it is displayed on a smartphone display or a head-mounted display. The input is avatar image data, and the output is the displayed avatar image.

[0652] Step 10:

[0653] The user checks the generated avatar image and sends any feedback they wish to provide to the server via their device. The input is the user's feedback, and the output is the feedback data sent to the server.

[0654] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0655] The present invention aims to generate natural and realistic avatar images that reflect the user's emotions by combining an emotion engine with a system that generates avatars from voice data. This system is implemented using the following components and procedures.

[0656] System Overview

[0657] The system includes the following major components:

[0658] Voice input means

[0659] A means of transmitting audio data

[0660] A means of preprocessing audio data

[0661] A means of extracting speech features

[0662] Emotion Engine

[0663] A method for selecting avatar models based on speech features and emotions

[0664] Avatar model and means for generating images based on emotions

[0665] A way to check and adjust the quality of the generated avatars

[0666] A means for transmitting the generated avatar image data

[0667] A means of displaying an avatar image

[0668] A way to receive user feedback

[0669] Program processing

[0670] The program processing in this system will be explained in natural language below.

[0671] 1. Audio input

[0672] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Sato. Please create a lively anime character."

[0673] 2. Sending audio data

[0674] The device records the user's voice and generates digital audio data, which is saved in a common audio file format (e.g., WAV, MP3) and sent to a server.

[0675] 3. Preprocessing of audio data

[0676] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[0677] The audio data is then converted into a digital format that makes it easier to analyze.

[0678] 4. Extraction of speech features

[0679] The server extracts features from the preprocessed speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0680] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[0681] 5. Emotional Recognition

[0682] The emotion engine recognizes the user's emotions from the voice data. The emotion engine analyzes voice parameters and distinguishes emotions such as joy, sadness, anger, and surprise.

[0683] Machine learning algorithms (e.g., neural networks, RNNs) are used to identify emotions from a pre-trained dataset.

[0684] 6. Select an avatar model

[0685] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character.

[0686] To select a model, a machine learning algorithm is used to select the optimal model from a pre-trained database of voice-character pairs.

[0687] 7. Image Generation

[0688] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[0689] 8. Avatar quality check and adjustment

[0690] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[0691] 9. Sending and displaying avatar images

[0692] The server generates the final avatar image data and sends it to the user's device, where it is saved in high-resolution format and provided to the user.

[0693] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[0694] Specific examples

[0695] Use case 1: Generating anime-style avatars that reflect emotions

[0696] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[0697] 2. The device records the audio and sends it to the server.

[0698] 3. The server preprocesses the audio data, removes noise, and extracts features.

[0699] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[0700] 5. The server selects a lively anime character model based on the features and emotions.

[0701] 6. The server uses GAN to generate an animated avatar with a happy expression.

[0702] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[0703] 8. The server sends the avatar image to the device.

[0704] 9. The device displays the avatar to the user, who confirms it.

[0705] The system allows users to quickly generate natural, high-quality, and emotionally sensitive avatars based on their voice.

[0706] The processing flow will be explained below.

[0707] Step 1:

[0708] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Suzuki. Please create a lively anime character."

[0709] Step 2:

[0710] The device records the user's voice and generates digital audio data that is stored in common audio file formats such as WAV or MP3.

[0711] Step 3:

[0712] The device sends the recorded audio data to the server, using an internet connection to transfer the data in a reliable manner (e.g. HTTPS).

[0713] Step 4:

[0714] The server receives the audio data and first performs noise reduction, using digital signal processing techniques (e.g. bandpass filters) to filter out background noise and unwanted sounds.

[0715] Step 5:

[0716] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are analyzed using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[0717] Step 6:

[0718] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes voice parameters to identify emotions such as joy, sadness, anger, and surprise. Specifically, it uses a machine learning algorithm (e.g., neural network, RNN) to identify emotions from a pre-trained dataset.

[0719] Step 7:

[0720] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character. The model selection uses a machine learning algorithm and references a pre-trained voice-character pair database.

[0721] Step 8:

[0722] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[0723] Step 9:

[0724] The server checks the quality of the generated avatar, specifically checking for facial distortions and unnatural expressions, and automatically corrects any unnatural features using post-processing algorithms.

[0725] Step 10:

[0726] The server sends the final avatar image data to the device, where it is stored in high-resolution format and may be compressed depending on the requirements of the service being used.

[0727] Step 11:

[0728] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything they don't like about it. For example, the user can make a specific request, such as "I want the facial expression to be more smiling."

[0729] Specific examples

[0730] Use case 1: Generating anime-style avatars that reflect emotions

[0731] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[0732] 2. The device records the audio and sends it to the server.

[0733] 3. The server preprocesses the audio data, removes noise, and extracts features.

[0734] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[0735] 5. The server selects a lively anime character model based on the features and emotions.

[0736] 6. The server uses GAN to generate an animated avatar with a happy expression.

[0737] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[0738] 8. The server sends the avatar image to the device.

[0739] 9. The device displays the avatar to the user, who confirms it.

[0740] Example 2

[0741] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] Conventional avatar generation systems have had difficulty generating natural and realistic avatar images that reflect the user's emotions. A particular technical challenge has been properly recognizing emotions from voice data and creating an avatar with corresponding facial expressions. Therefore, a system that can easily generate avatars that accurately reflect the user's intentions and emotions has been sought.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0744] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, means for recognizing emotions using an emotion engine, means for selecting an avatar model based on the voice features and emotions, means for generating an image based on the avatar model and emotions, means for checking and adjusting the quality of the generated avatar, and means for transmitting the generated avatar image data. This makes it possible to recognize a user's emotions from voice data and quickly generate natural and realistic avatar images that reflect those emotions.

[0745] The "voice input means" is a means that enables a user to utter voice data into the microphone of the terminal.

[0746] The "means for transmitting audio data" refers to a means for transferring recorded audio data to a server in digital format.

[0747] "Means for preprocessing audio data" refers to means for removing noise and filtering audio data, and converting the audio data into a format that is easy to analyze.

[0748] The "means for extracting speech features" refers to a means for extracting features such as pitch, formant, tone, and rhythm from speech data.

[0749] The "emotion engine" is an engine that uses machine learning algorithms to analyze the user's emotions from voice data and identify emotions such as joy, sadness, anger, and surprise.

[0750] The "means for selecting an avatar model" is a means for selecting an optimal avatar model based on speech features and emotions.

[0751] The "means for generating an image" is a means using a deep learning model to generate an avatar image based on a selected avatar model and a recognized emotion.

[0752] The "means for checking and adjusting the quality of the generated avatar" refers to a means for checking the quality of the generated avatar image and adjusting facial distortions and unnatural expressions using a post-processing algorithm.

[0753] The "means for transmitting generated avatar image data" refers to a means for transmitting the final avatar image data to a user's device.

[0754] The "means for displaying an avatar image" refers to a means for displaying the received avatar image on the user's terminal.

[0755] The "means for receiving user feedback" is a means for a user to provide opinions or requests for corrections to the generated avatar image.

[0756] The present invention is a system that recognizes a user's emotions from voice data and generates a natural and realistic avatar image that reflects those emotions. This system is composed of multiple hardware and software components. Specifically, it is implemented using the following components and procedures.

[0757] Main components

[0758] Voice input method: The user speaks into the device's microphone, describing the desired avatar's characteristics, messages, and emotions. The device captures the voice using a microphone sound input library (e.g., PyAudio).

[0759] How audio data is transmitted: The device saves the recorded audio data in a common audio file format (e.g., WAV, MP3) and transmits it to the server using an HTTP request.

[0760] A means of pre-processing the audio data: The server uses an audio processing library (e.g., Librosa) to denoise and filter the audio data, then convert it into a digital format that is easier to analyze.

[0761] Method for extracting speech features: The server uses FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formant, tone, and rhythm from the speech data.

[0762] Emotion engine: The server uses an emotion engine (e.g., neural network or RNN) to analyze the user's emotions from the voice features and identify emotions such as joy, sadness, anger, and surprise.

[0763] Method for selecting an avatar model: The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on audio features and emotion information.

[0764] Means for generating images: The server uses GAN (generative adversarial network) or StyleGAN to generate avatar images based on the selected avatar model and recognized emotions.

[0765] A means to check and adjust the quality of the generated avatar: The server checks the generated avatar image for facial distortion and unnatural expressions, and performs post-processing using image processing algorithms (e.g., OpenCV) to make any necessary fine-tuning.

[0766] Means for sending the generated avatar image data: The server saves the final avatar image data in a high-resolution format (e.g., PNG) and sends it to the user's device.

[0767] A method for displaying the avatar image: The device displays the received avatar image. For example, HTML5 Use tags to display in the browser.

[0768] A means of receiving user feedback: Users can provide specific opinions and requests for corrections to the displayed avatar image. For example, they can send feedback via their device, such as "I want the facial expression to be more smiling."

[0769] Usage example

[0770] Specific examples are shown below.

[0771] Use case 1: Generating anime-style avatars that reflect emotions

[0772] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[0773] 2. The device records the audio and sends it to the server in WAV format.

[0774] 3. The server uses Librosa to denoise and filter the audio data, making it easier to analyze.

[0775] 4. The server extracts features from the audio data using FFT and cepstrum analysis.

[0776] 5. The emotion engine identifies the user's emotion "joy" from the voice.

[0777] 6. The server uses random forests to select a model of a lively anime character based on features and emotions.

[0778] 7. The server uses GAN to generate an animated avatar with a happy expression.

[0779] 8. The server checks the quality of the generated avatar using OpenCV and makes adjustments if necessary.

[0780] 9. The server sends the final avatar image to the device.

[0781] 10. The device displays the received image and the user confirms it.

[0782] Prompt Sentence Examples

[0783] Generate a new avatar image using the voice data of a user saying "Hello, I'm Suzuki. Please create a lively anime character." The emotion to use is "joy."

[0784] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0785] Step 1:

[0786] The user speaks into the device's microphone, including the desired avatar's characteristics, message, and emotion. For example, "Hello, I'm Sato. Please create a lively anime character." This voice is input as an analog signal.

[0787] Step 2:

[0788] The device uses a microphone audio input library (e.g., PyAudio) to capture the user's voice and save it in a digital format (WAV or MP3). The saved voice data is then output.

[0789] Step 3:

[0790] The device sends audio data to the server via an HTTP request (e.g., using Python's requests library). This operation inputs digital audio data and transfers it to the server.

[0791] Step 4:

[0792] The server denoises the audio data using an audio processing library such as Librosa, filtering out background noise and unwanted sounds. The input is digital audio data, and the output is clean audio data after noise reduction.

[0793] Step 5:

[0794] The server uses an audio processing library to convert the clean audio data into a more analyzable form. This calculation includes formatting in digital form. The converted audio data is output.

[0795] Step 6:

[0796] The server performs FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formants, tone, and rhythm. The input is preprocessed speech data, and the extracted speech features are output.

[0797] Step 7:

[0798] The emotion engine analyzes the user's emotions using a machine learning algorithm (e.g., neural network, RNN) based on the extracted voice features. The input is the voice features, and the output is the identified emotion (e.g., joy, sadness).

[0799] Step 8:

[0800] The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on the identified emotions and voice features. The input is the voice features and emotion information, and the selected avatar model is output.

[0801] Step 9:

[0802] The server generates an avatar image using a deep learning model (e.g., GAN, StyleGAN) based on the selected avatar model and the identified emotion. The input is the avatar model and emotion information, and the generated avatar image is output.

[0803] Step 10:

[0804] The server uses image processing algorithms (e.g., OpenCV) to check the quality of the generated avatar image and check for facial distortions and unnatural expressions. If necessary, post-processing is performed and a fine-tuned avatar image is output.

[0805] Step 11:

[0806] The server saves the final avatar image in a high-resolution format (e.g., PNG) and sends it to the user's device as an HTTP response. The input is the adjusted avatar image data, and the output is the sent image data.

[0807] Step 12:

[0808] The device receives an avatar image and sends it to HTML5. It displays the image to the user using tags, etc. The user can review the image and provide specific feedback (e.g., "Please smile more") if there is anything they don't like. The input is avatar image data, and the displayed avatar image is output.

[0809] (Application example 2)

[0810] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0811] Conventional avatar generation systems have had difficulty generating avatars that accurately reflect a user's emotions in real time. In particular, live streaming and real-time content distribution require immediate reflection of the user's emotions and voice characteristics. To solve this problem, a system is needed that performs highly accurate emotion analysis and voice feature extraction, selects an avatar model based on the results, and generates images in real time.

[0812] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0813] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, and means for analyzing emotions, which enable real-time analysis of emotions from a user's voice, selection of an avatar model based on the emotions, and generation of an image, thereby enabling the generation of avatars that accurately reflect the user's emotions in live streaming or real-time content distribution.

[0814] The "voice input means" is a device or mechanism for acquiring the user's voice as digital voice data.

[0815] The "means for transmitting voice data" is a mechanism for transmitting the acquired voice data to another device or server.

[0816] "Means for preprocessing audio data" refers to a processing device or method for removing noise from audio data and making it easier to analyze.

[0817] A "means for extracting audio features" is a device or method for extracting characteristic data such as formants, pitch, tone, and rhythm from audio data.

[0818] "Means for analyzing emotions" refers to devices or methods for identifying and determining a user's emotions from voice data.

[0819] The "means for selecting an avatar model based on speech features and emotions" refers to a device or method for selecting an optimal avatar model based on the acquired speech features and analyzed emotions.

[0820] The "means for generating an image based on an avatar model and an emotion" refers to a device or method for generating a specific avatar image according to a selected avatar model and an analyzed emotion.

[0821] "Means for checking and adjusting the quality of the generated avatar" refers to a device or method for inspecting the generated avatar image for unnatural features or defects and correcting them if necessary.

[0822] The "means for transmitting the generated avatar image data" is a mechanism for transmitting the generated avatar image data to a user's terminal or another system.

[0823] The "means for displaying an avatar image" refers to a display device or a display program for visually displaying the generated avatar image.

[0824] The "means for receiving user feedback" refers to a device or interface for obtaining user opinions and impressions regarding the generated avatar image.

[0825] This invention relates to an avatar generation system that combines voice data and emotion analysis. The purpose of this system is to generate and display avatars in real time that reflect the user's emotions in real-time live streaming and content distribution.

[0826] Main system components

[0827] The system requires the following major components:

[0828] Voice input means

[0829] A means of transmitting audio data

[0830] A means of preprocessing audio data

[0831] A means of extracting speech features

[0832] A means of analyzing emotions

[0833] A method for selecting avatar models based on speech features and emotions

[0834] Avatar model and means for generating images based on emotions

[0835] A way to check and adjust the quality of the generated avatars

[0836] A means for transmitting the generated avatar image data

[0837] A means of displaying an avatar image

[0838] A way to receive user feedback

[0839] Hardware and software used

[0840] This system requires high-performance hardware and advanced software. The main hardware and software used are described below.

[0841] Hardware

[0842] Smartphone: An iOS or Android device used to input voice and display results.

[0843] High-performance server: Equipped with a GPU, it performs data preprocessing, voice analysis, emotion analysis, and avatar generation.

[0844] software

[0845] Audio processing libraries: e.g. librosa, used for preprocessing audio data (noising, filtering).

[0846] Machine learning frameworks: e.g., TensorFlow, PyTorch, used for audio feature extraction and sentiment analysis.

[0847] Deep learning models: e.g. GAN (generative artificial network), StyleGAN, used to generate avatar images.

[0848] Speech recognition models, such as DeepSpeech, are used to extract features from speech data.

[0849] Data processing flow

[0850] The server first receives the voice data and performs preprocessing. Next, it extracts voice features and analyzes emotions. Based on this information, it selects an avatar model and generates an avatar image in real time. The generated image undergoes a quality check and is then sent to the user's smartphone. The smartphone displays the avatar image and receives user feedback.

[0851] Specific examples

[0852] Example 1: Real-time avatar generation during live streaming

[0853] When a user starts live streaming on their smartphone, they begin speaking into the voice input means. For example, they might say, "Hello, I'm so happy today!" This speech is sent to the server in real time. The server preprocesses the speech data and extracts features to analyze emotions. Based on the analysis results, an avatar model is selected and an avatar image is generated in real time. This avatar image is sent to the smartphone and visually displayed to the user.

[0854] Prompt Sentence Examples

[0855] "Hello, I'm so happy today!"

[0856] => Emotion: Joy

[0857] => Avatar Model: Smiling Anime Character

[0858] This detailed processing procedure generates a high-quality avatar that reflects the user's emotions in real time, enabling viewers to experience immersive live streaming.

[0859] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0860] Step 1:

[0861] The user begins speaking into the smartphone's microphone. The input is the user's voice data, for example, "Hello, I'm very happy today!" The output is recorded digital voice data. The device transmits this voice data to the server in real time.

[0862] Step 2:

[0863] The server receives the audio data sent from the terminal. The input is the digital audio data sent from the terminal. The server first preprocesses this audio data and performs noise removal. As a result, clean audio data with noise removed is obtained as output.

[0864] Step 3:

[0865] The server extracts features from the preprocessed speech data. The input is clean speech data with noise removed. Specifically, it uses FFT (Fast Fourier Transform) and cepstrum analysis to extract characteristic data such as formants, pitch, tone, and rhythm. The output is these speech feature data.

[0866] Step 4:

[0867] The server performs emotion analysis based on the speech feature data. The input is speech feature data. Neural networks or RNNs are used for emotion analysis to analyze emotions such as joy, sadness, anger, and surprise. The output is the analyzed emotion data.

[0868] Step 5:

[0869] The server selects the optimal avatar model based on the extracted speech features and analyzed emotion data. The input is speech feature data and emotion data. The server uses a machine learning algorithm to select the optimal avatar model from a pre-trained database. The output is the selected avatar model.

[0870] Step 6:

[0871] The server generates an avatar image based on the selected avatar model and the analyzed emotion data. The input is the avatar model and emotion data. The server uses a deep learning model (e.g., GAN, StyleGAN) to generate the avatar's face and expressions. The output is the generated avatar image.

[0872] Step 7:

[0873] The server checks the quality of the generated avatar image. The input is the generated avatar image. The server checks for facial distortion and unnatural expressions, and makes adjustments using post-processing algorithms if necessary. The output is the final quality-assured avatar image.

[0874] Step 8:

[0875] The server sends the final avatar image data to the user's smartphone. The input is a quality-guaranteed avatar image. The output is the avatar image displayed on the smartphone.

[0876] Step 9:

[0877] The device visually displays the received avatar image to the user. The input is the avatar image data sent from the server. The user can review the avatar image and provide feedback. The feedback is sent to the system through a robust user interface and used for further improvement.

[0878] Specific prompt examples:

[0879] "Hello, I'm so happy today!"

[0880] => Emotion: Joy

[0881] => Avatar Model: Smiling Anime Character

[0882] This processing step generates high-quality avatar images that reflect the user's emotions in real time, providing a more lifelike experience for the user and viewers.

[0883] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0884] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0885] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0886] [Third embodiment]

[0887] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0888] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0889] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0890] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0891] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0892] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0893] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0894] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0895] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0896] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0897] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0898] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0899] The present invention provides a system for generating avatars from voice data. The system aims to generate natural and realistic avatar images from a user's voice input, and is implemented by the following method and procedure.

[0900] System Overview

[0901] The system includes the following major components:

[0902] Voice input means

[0903] A means of transmitting audio data

[0904] A means of preprocessing audio data

[0905] A means of extracting speech features

[0906] A method for selecting avatar models based on speech features

[0907] A method for generating images based on avatar models

[0908] A way to check and adjust the quality of the generated avatars

[0909] A means for transmitting the generated avatar image data

[0910] A means of displaying an avatar image

[0911] A way to receive user feedback

[0912] Program processing

[0913] The program processing in this system will be explained in natural language below.

[0914] 1. Audio input

[0915] The user speaks into the device, and can include the message they want to convey and the characteristics of the avatar they want to create. For example, they might say, "Hello, my name is Sato. Please create an anime character for me."

[0916] 2. Sending audio data

[0917] The terminal records the user's voice and transmits the digital voice data to the server.

[0918] 3. Preprocessing of audio data

[0919] The server receives the audio data and first performs noise reduction, which filters out background noise and unwanted sounds.

[0920] The audio data is then converted into a digital format that makes it easier to analyze.

[0921] 4. Extraction of speech features

[0922] The server extracts speech features from the speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0923] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[0924] 5. Select an avatar model

[0925] The server selects the most suitable avatar model based on the extracted voice features. For example, if the voice is bright and high-pitched, it selects a youthful anime character.

[0926] To select a model, a machine learning algorithm is used, referencing pre-trained voice-character pair data.

[0927] 6. Image Generation

[0928] The server generates an image based on the selected avatar model, specifically using deep learning models such as GAN (Generative Adversarial Network) and StyleGAN.

[0929] This generates a facial pattern corresponding to the audio features.

[0930] 7. Avatar quality check and adjustment

[0931] The server checks the quality of the generated avatar, checking for unnatural features such as facial distortion or unnatural facial expressions.

[0932] If necessary, fine-tune using post-processing algorithms.

[0933] 8. Sending and displaying avatar images

[0934] The server generates the final avatar image data and sends it to the user's terminal.

[0935] The device displays the received avatar image to the user, who can then review the avatar image and provide feedback if necessary.

[0936] Specific examples

[0937] Below are some specific usage examples.

[0938] Use case 1: Anime-style avatar generation

[0939] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create an anime character for me."

[0940] 2. The device records the audio and sends it to the server.

[0941] 3. The server preprocesses the audio data, removes noise, and extracts features.

[0942] 4. The server selects a youthful anime character model based on the features.

[0943] 5. The server uses GAN to generate an animated avatar.

[0944] 6. The server checks the quality of the generated avatar and makes adjustments if necessary.

[0945] 7. The server sends the avatar image to the device.

[0946] 8. The device displays the avatar to the user, who confirms it.

[0947] This system allows users to quickly generate a natural voice-based avatar to enhance visual communication.

[0948] The processing flow will be explained below.

[0949] Step 1:

[0950] The user speaks into the device's microphone, describing the desired avatar's characteristics and a message. For example, they might say, "Hello, I'm Sato. Please create an anime character for me."

[0951] Step 2:

[0952] The device records the user's voice and generates digital audio data that is stored in a common audio file format (e.g., WAV, MP3).

[0953] Step 3:

[0954] The device sends the recorded audio data to a server, using an internet connection for data transfer.

[0955] Step 4:

[0956] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[0957] Step 5:

[0958] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are extracted using FFT (Fast Fourier Transform) and cepstrum analysis.

[0959] Step 6:

[0960] The server selects an appropriate avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest, SVM) to select the best model from a pre-trained voice-character pair database.

[0961] Step 7:

[0962] The server generates an avatar image based on the selected avatar model, and uses a deep learning model (e.g., GAN, StyleGAN) to generate a facial pattern corresponding to the audio features.

[0963] Step 8:

[0964] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[0965] Step 9:

[0966] The server sends the final avatar image data to the device, where it is saved in high-resolution format and provided to the user.

[0967] Step 10:

[0968] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[0969] This series of steps allows users to quickly generate a natural, high-quality avatar based on their voice.

[0970] Example 1

[0971] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0972] Conventional avatar generation systems have had many problems when generating natural and realistic avatars from user voice, particularly in the preprocessing of voice data, feature extraction, and final image generation and quality adjustment. Specifically, the efficiency of noise removal from voice data was low, and the accuracy of extracted features was insufficient, making it often impossible to select the optimal avatar model. Furthermore, the quality of the generated images was unstable, with many unnatural parts. A method to resolve these issues is needed.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0974] In this invention, the server includes means for performing FFT analysis on the voice data, means for generating avatar images using GAN, and means for adjusting the quality of the generated images using a post-processing algorithm, which enables highly accurate analysis and noise removal of the voice data, selection of the optimal avatar model, and generation and adjustment of high-quality avatar images.

[0975] "Voice input means" refers to a device or interface that allows a user to input voice, such as a microphone or voice recognition software.

[0976] "Means for transmitting audio data" refers to the communications means for transmitting audio data from the user's device to the server, typically including an internet connection and the HTTP protocol.

[0977] "Means for pre-processing audio data" refers to processes for converting recorded audio data into an analyzable format, including, for example, noise reduction and digital conversion.

[0978] "Means for extracting speech features" refers to techniques for analyzing and extracting necessary speech features from preprocessed speech data, including FFT (Fast Fourier Transform) and cepstrum analysis.

[0979] "Means for selecting an avatar model based on speech features" refers to an algorithm for selecting an optimal avatar model based on extracted speech features, including machine learning algorithms.

[0980] "Means for generating images based on an avatar model" refers to technology for generating the final avatar image based on the selected avatar model, including deep learning models such as Generative Adversarial Networks (GANs) and StyleGANs.

[0981] "Means for checking and adjusting the quality of the generated avatar" refers to technology, including post-processing algorithms, for checking whether the generated avatar image is appropriate and correcting it if necessary.

[0982] "Means for transmitting generated avatar image data" refers to a communication means for transmitting the generated avatar image from the server to the user's device, such as the HTTP protocol.

[0983] "Means for displaying an avatar image" refers to technology for displaying the received avatar image on a user's device, including a display or monitor.

[0984] "Means for receiving user feedback" refers to an interface for collecting user opinions and comments about the generated avatar, including a GUI or web form.

[0985] "FFT analysis" refers to the fast Fourier transform technology used to efficiently analyze the frequency components of audio data.

[0986] "Means for generating avatar images using GAN" refers to technology for automatically generating natural-looking avatar images using a generative artificial network (GAN).

[0987] "Post-processing algorithms" refers to post-processing techniques that apply additional adjustments and modifications to the generated images, including quality improvement using OpenCV.

[0988] MODE FOR CARRYING OUT THE INVENTION

[0989] The present invention is a system that generates realistic and natural avatar images based on voice input. The series of processes that generate an avatar from a user's voice input and display that image on a terminal will be described below.

[0990] First, the user speaks into the device. This voice input is collected using a microphone and voice recognition software. For example, the user might give a voice command such as, "Hello, I'm Suzuki. Please create an anime character for me." This voice data is then digitally recorded on the device.

[0991] The device then sends the recorded audio data to the server using an HTTP POST request over the internet connection, in a common audio file format such as WAV.

[0992] The server performs preprocessing on the received audio data. This preprocessing includes noise reduction, which uses filters to remove background noise and unwanted sounds to clean the audio data. It also analyzes the frequency components of the audio data using FFT (Fast Fourier Transform). This allows it to extract audio features such as pitch (high and low pitch), formants (resonant frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[0993] Based on the extracted features, the server selects the optimal avatar model. This selection is performed using a machine learning algorithm, such as a random forest or deep neural network, to match the voice features with the avatar model. If the user's voice has a youthful tone, a youthful anime character model will be selected.

[0994] After the model is selected, the server generates an avatar image using a generative AI model such as a Generative Adversarial Network (GAN) or StyleGAN. This process generates a natural-looking avatar that reflects the audio features. The generated avatar image is then quality checked and adjusted using post-processing algorithms. For example, libraries such as OpenCV are used to correct unnatural facial distortions and imperfections in facial expressions.

[0995] The final avatar image is sent from the server to the device. Again, the HTTP protocol is generally used. The device displays the received avatar image to the user. The user can check the displayed avatar and provide feedback if necessary. This feedback is reflected in the next avatar generation, enabling the generation of a higher quality avatar.

[0996] Specific examples

[0997] Use case 1: Anime-style avatar generation

[0998] Example prompt sentence:

[0999] "Hello, I'm Suzuki. Please make me an anime character."

[1000] The system allows users to quickly generate natural voice-based avatars to enhance visual communication, a process achieved through the use of highly accurate voice analysis and generative AI models.

[1001] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1002] Program processing flow

[1003] Step 1:

[1004] The user speaks to the device, for example, "Hello, my name is Sato. Please create an anime character for me." At this time, the voice input is recorded as digital voice data via a microphone.

[1005] Input: User's voice

[1006] Output: Digital audio data (WAV, etc.)

[1007] Step 2:

[1008] The device sends the recorded audio data in digital format to the server using an HTTP POST request.

[1009] Input: Digital audio data

[1010] Output: Audio data sent in HTTP POST request

[1011] Step 3:

[1012] The server first denoises the received audio data to convert it into an analyzable format, using a filter to remove background noise and obtain clean audio data, then resamples the audio data to a standard sampling rate.

[1013] Input: Transmitted audio data

[1014] Output: Preprocessed, clean audio data

[1015] Step 4:

[1016] The server analyzes the frequency components from the preprocessed audio data using FFT (Fast Fourier Transform), extracting audio features such as pitch, formants, tone, and rhythm. The features are stored in list or vector format.

[1017] Input: Preprocessed clean audio data

[1018] Output: A list or vector of audio features

[1019] Step 5:

[1020] The server selects the optimal avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest or deep neural network) and references a database of voice features and avatar models. The selected model is recorded as the selection result.

[1021] Input: List of audio features

[1022] Output: Selected avatar model

[1023] Step 6:

[1024] The server generates an avatar image based on the selected avatar model using a generative AI model (e.g., GAN or StyleGAN). The generation process uses multiple neural networks to generate the final avatar image.

[1025] Input: Selected avatar model

[1026] Output: Generated avatar image

[1027] Step 7:

[1028] The server performs a quality check on the generated avatar images, detecting any unnatural features (facial distortions or imperfections in expressions) and correcting them using post-processing algorithms (e.g., OpenCV).

[1029] Input: Generated avatar image

[1030] Output: Quality adjusted avatar image

[1031] Step 8:

[1032] The server then sends the final avatar image data to the user's device, again using the HTTP protocol.

[1033] Input: Quality adjusted avatar image

[1034] Output: Avatar image data sent in the HTTP response

[1035] Step 9:

[1036] The device displays the received avatar image to the user. The avatar image can be visually confirmed using the display. The user can check the displayed avatar image and provide feedback through the device.

[1037] Input: Avatar image data received in the HTTP response

[1038] Output: Avatar image displayed on the screen

[1039] Through this series of steps, users can quickly generate realistic and natural avatar images based on voice input, enhancing visual communication.

[1040] (Application example 1)

[1041] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1042] In conventional virtual stores, there has been no system that provides an assistant avatar that can interact with users in real time. This has led to the issue of not being able to improve the interactive shopping experience in stores. There has also been a lack of systems that can generate and display an appropriate avatar based on the user's voice request.

[1043] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1044] In this invention, the server includes a voice input means, a voice data transmission means, a voice data preprocessing means, a voice feature extraction means, a voice feature selection means, an avatar model selection means based on the voice feature, an image generation means based on the avatar model, a quality check and adjustment means for the generated avatar, a transmission means for the generated avatar image data, a display means for the avatar image, a user feedback receiving means, and a generation means for an assistant avatar to provide an interactive shopping experience in a virtual store. This allows a user to easily generate an avatar through voice input and realize an interactive shopping experience.

[1045] "Voice input means" refers to a device for capturing the user's voice, and includes a microphone and the voice input function of a smartphone.

[1046] "Means for transmitting audio data" refers to a communications interface or network protocol for transmitting recorded audio data to a server or other processing system.

[1047] "Audio data preprocessing means" refers to algorithms or software that remove noise from recorded audio data and convert the data into a format that is easier to analyze.

[1048] "Means for extracting speech features" refers to technology for extracting characteristic elements of speech, such as pitch, formants, tone, and rhythm, from speech data.

[1049] The "means for selecting an avatar model based on speech features" refers to a machine learning algorithm or database for selecting the optimal avatar model based on the extracted speech features.

[1050] "Means for generating images based on an avatar model" refers to technologies for generating actual avatar images from a selected avatar model, and includes deep learning technologies such as GAN (generative adversarial networks) and StyleGAN.

[1051] "Means for checking and adjusting the quality of the generated avatar" refers to algorithms or software for evaluating the quality of the generated avatar image and correcting any unnatural parts.

[1052] The "means for transmitting generated avatar image data" refers to a communication interface or network protocol for transmitting the avatar image data generated by the server to a user's device.

[1053] The "means for displaying an avatar image" refers to a display device or application that a user uses to view the generated avatar image.

[1054] "Means for receiving user feedback" refers to an interface or system for collecting evaluations and opinions from users.

[1055] "Means for generating assistant avatars to provide an interactive shopping experience within a virtual store" refers to a technology that generates an avatar based on voice data to interact with users in a virtual store environment and provide guidance and support.

[1056] A system for implementing the present invention includes a voice input means, a means for transmitting voice data, a means for preprocessing the voice data, a means for extracting voice features, a means for selecting an avatar model based on the voice features, a means for generating an image based on the avatar model, a means for checking and adjusting the quality of the generated avatar, a means for transmitting the generated avatar image data, a means for displaying the avatar image, and a means for receiving user feedback.

[1057] First, the user uses a voice input means such as a smartphone or a head-mounted display to input a voice such as "Hello, I'm looking for shoes." The voice data is recorded through the microphone of the device and transmitted to the server via the network by a means for transmitting voice data.

[1058] The server preprocesses the received audio data, removing noise and converting it to digital form. Specifically, it uses the Python speech_recognition library. From the preprocessed audio data, the server extracts audio features. Using FFT (Fast Fourier Transform) and cepstrum analysis, audio features such as pitch, formant, tone, and rhythm are extracted.

[1059] Based on the extracted voice features, the server selects the optimal avatar model using machine learning algorithms and pre-trained voice-character pair data. For example, if the voice tone is bright and high, a youthful anime character model will be selected at this stage.

[1060] Based on the selected avatar model, the server generates an avatar image using deep learning techniques (e.g., GAN or StyleGAN). The generated avatar image is then quality-evaluated by the server to ensure there are no unnatural features. If there are any defects, they are corrected using a post-processing algorithm.

[1061] The final avatar image data after adjustment is sent from the server to the device. The device displays the received avatar image on the user's display device. The user can review the avatar and provide feedback if necessary. This feedback is sent to the server and is used to improve the system and increase the accuracy of avatar generation.

[1062] As a specific example of use, consider a situation where a user speaks to a device, "Hello, I'm Suzuki. Please create an anime character for me." The server selects an avatar model from this voice data, generates an anime-style avatar using GAN, and finally displays it on the device as a series of processes.

[1063] Another example of a prompt when generating an avatar using a generative AI model is, "Please generate an avatar that is a youthful female character with a friendly smile."

[1064] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1065] Step 1:

[1066] The user inputs voice into the device. This voice input includes the characteristics and wishes of the avatar the user wants to create. For example, the user might say, "Hello, my name is Sato. Please create an anime character for me." The voice data is recorded via a smartphone or microphone. The input is voice data, and the output is a recorded voice file.

[1067] Step 2:

[1068] The device sends the recorded audio data to the server, using communication methods such as the Internet or Bluetooth. The input is an audio file, and the output is the transmission of audio data to the server.

[1069] Step 3:

[1070] The server preprocesses the received audio data. Specifically, it performs noise removal and digital conversion of the audio signal. This preprocessing uses the speech_recognition library and other audio processing algorithms. The input is audio data, and the output is the preprocessed audio data.

[1071] Step 4:

[1072] The server extracts features from the speech data. For example, it uses FFT (Fast Fourier Transform) and cepstrum analysis techniques to extract speech features such as pitch, formant, tone, and rhythm. The input is preprocessed speech data, and the output is speech features.

[1073] Step 5:

[1074] The server selects an avatar model based on the extracted voice features. Here, a machine learning algorithm is used to compare the extracted voice features with pre-trained data and select the optimal avatar model. The input is the voice features, and the output is the selected avatar model.

[1075] Step 6:

[1076] The server generates an image based on the selected avatar model. Specifically, it uses a generative AI model such as GAN or StyleGAN to generate the avatar image. It uses a prompt such as "Please generate an avatar that is a youthful female character with a friendly smile." The input is the avatar model, and the output is the generated avatar image data.

[1077] Step 7:

[1078] The server checks the quality of the generated avatar and adjusts any unnatural parts. Quality assessment algorithms and post-processing algorithms are used here. The input is the generated avatar image data, and the output is the adjusted avatar image data.

[1079] Step 8:

[1080] The server sends the adjusted avatar image data to the user's device. Data is transferred using a network protocol. The input is the adjusted avatar image data, and the output is data sent to the device.

[1081] Step 9:

[1082] The device displays the received avatar image. Specifically, it is displayed on a smartphone display or a head-mounted display. The input is avatar image data, and the output is the displayed avatar image.

[1083] Step 10:

[1084] The user checks the generated avatar image and sends any feedback they wish to provide to the server via their device. The input is the user's feedback, and the output is the feedback data sent to the server.

[1085] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1086] The present invention aims to generate natural and realistic avatar images that reflect the user's emotions by combining an emotion engine with a system that generates avatars from voice data. This system is implemented using the following components and procedures.

[1087] System Overview

[1088] The system includes the following major components:

[1089] Voice input means

[1090] A means of transmitting audio data

[1091] A means of preprocessing audio data

[1092] A means of extracting speech features

[1093] Emotion Engine

[1094] A method for selecting avatar models based on speech features and emotions

[1095] Avatar model and means for generating images based on emotions

[1096] A way to check and adjust the quality of the generated avatars

[1097] A means for transmitting the generated avatar image data

[1098] A means of displaying an avatar image

[1099] A way to receive user feedback

[1100] Program processing

[1101] The program processing in this system will be explained in natural language below.

[1102] 1. Audio input

[1103] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Sato. Please create a lively anime character."

[1104] 2. Sending audio data

[1105] The device records the user's voice and generates digital audio data, which is saved in a common audio file format (e.g., WAV, MP3) and sent to a server.

[1106] 3. Preprocessing of audio data

[1107] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[1108] The audio data is then converted into a digital format that makes it easier to analyze.

[1109] 4. Extraction of speech features

[1110] The server extracts features from the preprocessed speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[1111] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[1112] 5. Emotional Recognition

[1113] The emotion engine recognizes the user's emotions from the voice data. The emotion engine analyzes voice parameters and distinguishes emotions such as joy, sadness, anger, and surprise.

[1114] Machine learning algorithms (e.g., neural networks, RNNs) are used to identify emotions from a pre-trained dataset.

[1115] 6. Select an avatar model

[1116] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character.

[1117] To select a model, a machine learning algorithm is used to select the optimal model from a pre-trained database of voice-character pairs.

[1118] 7. Image Generation

[1119] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[1120] 8. Avatar quality check and adjustment

[1121] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[1122] 9. Sending and displaying avatar images

[1123] The server generates the final avatar image data and sends it to the user's device, where it is saved in high-resolution format and provided to the user.

[1124] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[1125] Specific examples

[1126] Use case 1: Generating anime-style avatars that reflect emotions

[1127] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[1128] 2. The device records the audio and sends it to the server.

[1129] 3. The server preprocesses the audio data, removes noise, and extracts features.

[1130] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[1131] 5. The server selects a lively anime character model based on the features and emotions.

[1132] 6. The server uses GAN to generate an animated avatar with a happy expression.

[1133] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[1134] 8. The server sends the avatar image to the device.

[1135] 9. The device displays the avatar to the user, who confirms it.

[1136] The system allows users to quickly generate natural, high-quality, and emotionally sensitive avatars based on their voice.

[1137] The processing flow will be explained below.

[1138] Step 1:

[1139] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Suzuki. Please create a lively anime character."

[1140] Step 2:

[1141] The device records the user's voice and generates digital audio data that is stored in common audio file formats such as WAV or MP3.

[1142] Step 3:

[1143] The device sends the recorded audio data to the server, using an internet connection to transfer the data in a reliable manner (e.g. HTTPS).

[1144] Step 4:

[1145] The server receives the audio data and first performs noise reduction, using digital signal processing techniques (e.g. bandpass filters) to filter out background noise and unwanted sounds.

[1146] Step 5:

[1147] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are analyzed using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[1148] Step 6:

[1149] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes voice parameters to identify emotions such as joy, sadness, anger, and surprise. Specifically, it uses a machine learning algorithm (e.g., neural network, RNN) to identify emotions from a pre-trained dataset.

[1150] Step 7:

[1151] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character. The model selection uses a machine learning algorithm and references a pre-trained voice-character pair database.

[1152] Step 8:

[1153] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[1154] Step 9:

[1155] The server checks the quality of the generated avatar, specifically checking for facial distortions and unnatural expressions, and automatically corrects any unnatural features using post-processing algorithms.

[1156] Step 10:

[1157] The server sends the final avatar image data to the device, where it is stored in high-resolution format and may be compressed depending on the requirements of the service being used.

[1158] Step 11:

[1159] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything they don't like about it. For example, the user can make a specific request, such as "I want the facial expression to be more smiling."

[1160] Specific examples

[1161] Use case 1: Generating anime-style avatars that reflect emotions

[1162] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[1163] 2. The device records the audio and sends it to the server.

[1164] 3. The server preprocesses the audio data, removes noise, and extracts features.

[1165] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[1166] 5. The server selects a lively anime character model based on the features and emotions.

[1167] 6. The server uses GAN to generate an animated avatar with a happy expression.

[1168] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[1169] 8. The server sends the avatar image to the device.

[1170] 9. The device displays the avatar to the user, who confirms it.

[1171] Example 2

[1172] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1173] Conventional avatar generation systems have had difficulty generating natural and realistic avatar images that reflect the user's emotions. A particular technical challenge has been properly recognizing emotions from voice data and creating an avatar with corresponding facial expressions. Therefore, a system that can easily generate avatars that accurately reflect the user's intentions and emotions has been sought.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1175] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, means for recognizing emotions using an emotion engine, means for selecting an avatar model based on the voice features and emotions, means for generating an image based on the avatar model and emotions, means for checking and adjusting the quality of the generated avatar, and means for transmitting the generated avatar image data. This makes it possible to recognize a user's emotions from voice data and quickly generate natural and realistic avatar images that reflect those emotions.

[1176] The "voice input means" is a means that enables a user to utter voice data into the microphone of the terminal.

[1177] The "means for transmitting audio data" refers to a means for transferring recorded audio data to a server in digital format.

[1178] "Means for preprocessing audio data" refers to means for removing noise and filtering audio data, and converting the audio data into a format that is easy to analyze.

[1179] The "means for extracting speech features" refers to a means for extracting features such as pitch, formant, tone, and rhythm from speech data.

[1180] The "emotion engine" is an engine that uses machine learning algorithms to analyze the user's emotions from voice data and identify emotions such as joy, sadness, anger, and surprise.

[1181] The "means for selecting an avatar model" is a means for selecting an optimal avatar model based on speech features and emotions.

[1182] The "means for generating an image" is a means using a deep learning model to generate an avatar image based on a selected avatar model and a recognized emotion.

[1183] The "means for checking and adjusting the quality of the generated avatar" refers to a means for checking the quality of the generated avatar image and adjusting facial distortions and unnatural expressions using a post-processing algorithm.

[1184] The "means for transmitting generated avatar image data" refers to a means for transmitting the final avatar image data to a user's device.

[1185] The "means for displaying an avatar image" refers to a means for displaying the received avatar image on the user's terminal.

[1186] The "means for receiving user feedback" is a means for a user to provide opinions or requests for corrections to the generated avatar image.

[1187] The present invention is a system that recognizes a user's emotions from voice data and generates a natural and realistic avatar image that reflects those emotions. This system is composed of multiple hardware and software components. Specifically, it is implemented using the following components and procedures.

[1188] Main components

[1189] Voice input method: The user speaks into the device's microphone, describing the desired avatar's characteristics, messages, and emotions. The device captures the voice using a microphone sound input library (e.g., PyAudio).

[1190] How audio data is transmitted: The device saves the recorded audio data in a common audio file format (e.g., WAV, MP3) and transmits it to the server using an HTTP request.

[1191] A means of pre-processing the audio data: The server uses an audio processing library (e.g., Librosa) to denoise and filter the audio data, then convert it into a digital format that is easier to analyze.

[1192] Method for extracting speech features: The server uses FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formant, tone, and rhythm from the speech data.

[1193] Emotion engine: The server uses an emotion engine (e.g., neural network or RNN) to analyze the user's emotions from the voice features and identify emotions such as joy, sadness, anger, and surprise.

[1194] Method for selecting an avatar model: The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on audio features and emotion information.

[1195] Means for generating images: The server uses GAN (generative adversarial network) or StyleGAN to generate avatar images based on the selected avatar model and recognized emotions.

[1196] A means to check and adjust the quality of the generated avatar: The server checks the generated avatar image for facial distortion and unnatural expressions, and performs post-processing using image processing algorithms (e.g., OpenCV) to make any necessary fine-tuning.

[1197] Means for sending the generated avatar image data: The server saves the final avatar image data in a high-resolution format (e.g., PNG) and sends it to the user's device.

[1198] A method for displaying the avatar image: The device displays the received avatar image. For example, HTML5 Use tags to display in the browser.

[1199] A means of receiving user feedback: Users can provide specific opinions and requests for corrections to the displayed avatar image. For example, they can send feedback via their device, such as "I want the facial expression to be more smiling."

[1200] Usage example

[1201] Specific examples are shown below.

[1202] Use case 1: Generating anime-style avatars that reflect emotions

[1203] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[1204] 2. The device records the audio and sends it to the server in WAV format.

[1205] 3. The server uses Librosa to denoise and filter the audio data, making it easier to analyze.

[1206] 4. The server extracts features from the audio data using FFT and cepstrum analysis.

[1207] 5. The emotion engine identifies the user's emotion "joy" from the voice.

[1208] 6. The server uses random forests to select a model of a lively anime character based on features and emotions.

[1209] 7. The server uses GAN to generate an animated avatar with a happy expression.

[1210] 8. The server checks the quality of the generated avatar using OpenCV and makes adjustments if necessary.

[1211] 9. The server sends the final avatar image to the device.

[1212] 10. The device displays the received image and the user confirms it.

[1213] Prompt Sentence Examples

[1214] Generate a new avatar image using the voice data of a user saying "Hello, I'm Suzuki. Please create a lively anime character." The emotion to use is "joy."

[1215] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1216] Step 1:

[1217] The user speaks into the device's microphone, including the desired avatar's characteristics, message, and emotion. For example, "Hello, I'm Sato. Please create a lively anime character." This voice is input as an analog signal.

[1218] Step 2:

[1219] The device uses a microphone audio input library (e.g., PyAudio) to capture the user's voice and save it in a digital format (WAV or MP3). The saved voice data is then output.

[1220] Step 3:

[1221] The device sends audio data to the server via an HTTP request (e.g., using Python's requests library). This operation inputs digital audio data and transfers it to the server.

[1222] Step 4:

[1223] The server denoises the audio data using an audio processing library such as Librosa, filtering out background noise and unwanted sounds. The input is digital audio data, and the output is clean audio data after noise reduction.

[1224] Step 5:

[1225] The server uses an audio processing library to convert the clean audio data into a more analyzable form. This calculation includes formatting in digital form. The converted audio data is output.

[1226] Step 6:

[1227] The server performs FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formants, tone, and rhythm. The input is preprocessed speech data, and the extracted speech features are output.

[1228] Step 7:

[1229] The emotion engine analyzes the user's emotions using a machine learning algorithm (e.g., neural network, RNN) based on the extracted voice features. The input is the voice features, and the output is the identified emotion (e.g., joy, sadness).

[1230] Step 8:

[1231] The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on the identified emotions and voice features. The input is the voice features and emotion information, and the selected avatar model is output.

[1232] Step 9:

[1233] The server generates an avatar image using a deep learning model (e.g., GAN, StyleGAN) based on the selected avatar model and the identified emotion. The input is the avatar model and emotion information, and the generated avatar image is output.

[1234] Step 10:

[1235] The server uses image processing algorithms (e.g., OpenCV) to check the quality of the generated avatar image and check for facial distortions and unnatural expressions. If necessary, post-processing is performed and a fine-tuned avatar image is output.

[1236] Step 11:

[1237] The server saves the final avatar image in a high-resolution format (e.g., PNG) and sends it to the user's device as an HTTP response. The input is the adjusted avatar image data, and the output is the sent image data.

[1238] Step 12:

[1239] The device receives an avatar image and sends it to HTML5. It displays the image to the user using tags, etc. The user can review the image and provide specific feedback (e.g., "Please smile more") if there is anything they don't like. The input is avatar image data, and the displayed avatar image is output.

[1240] (Application example 2)

[1241] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1242] Conventional avatar generation systems have had difficulty generating avatars that accurately reflect a user's emotions in real time. In particular, live streaming and real-time content distribution require immediate reflection of the user's emotions and voice characteristics. To solve this problem, a system is needed that performs highly accurate emotion analysis and voice feature extraction, selects an avatar model based on the results, and generates images in real time.

[1243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1244] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, and means for analyzing emotions, which enable real-time analysis of emotions from a user's voice, selection of an avatar model based on the emotions, and generation of an image, thereby enabling the generation of avatars that accurately reflect the user's emotions in live streaming or real-time content distribution.

[1245] The "voice input means" is a device or mechanism for acquiring the user's voice as digital voice data.

[1246] The "means for transmitting voice data" is a mechanism for transmitting the acquired voice data to another device or server.

[1247] "Means for preprocessing audio data" refers to a processing device or method for removing noise from audio data and making it easier to analyze.

[1248] A "means for extracting audio features" is a device or method for extracting characteristic data such as formants, pitch, tone, and rhythm from audio data.

[1249] "Means for analyzing emotions" refers to devices or methods for identifying and determining a user's emotions from voice data.

[1250] The "means for selecting an avatar model based on speech features and emotions" refers to a device or method for selecting an optimal avatar model based on the acquired speech features and analyzed emotions.

[1251] The "means for generating an image based on an avatar model and an emotion" refers to a device or method for generating a specific avatar image according to a selected avatar model and an analyzed emotion.

[1252] "Means for checking and adjusting the quality of the generated avatar" refers to a device or method for inspecting the generated avatar image for unnatural features or defects and correcting them if necessary.

[1253] The "means for transmitting the generated avatar image data" is a mechanism for transmitting the generated avatar image data to a user's terminal or another system.

[1254] The "means for displaying an avatar image" refers to a display device or a display program for visually displaying the generated avatar image.

[1255] The "means for receiving user feedback" refers to a device or interface for obtaining user opinions and impressions regarding the generated avatar image.

[1256] This invention relates to an avatar generation system that combines voice data and emotion analysis. The purpose of this system is to generate and display avatars in real time that reflect the user's emotions in real-time live streaming and content distribution.

[1257] Main system components

[1258] The system requires the following major components:

[1259] Voice input means

[1260] A means of transmitting audio data

[1261] A means of preprocessing audio data

[1262] A means of extracting speech features

[1263] A means of analyzing emotions

[1264] A method for selecting avatar models based on speech features and emotions

[1265] Avatar model and means for generating images based on emotions

[1266] A way to check and adjust the quality of the generated avatars

[1267] A means for transmitting the generated avatar image data

[1268] A means of displaying an avatar image

[1269] A way to receive user feedback

[1270] Hardware and software used

[1271] This system requires high-performance hardware and advanced software. The main hardware and software used are described below.

[1272] Hardware

[1273] Smartphone: An iOS or Android device used to input voice and display results.

[1274] High-performance server: Equipped with a GPU, it performs data preprocessing, voice analysis, emotion analysis, and avatar generation.

[1275] software

[1276] Audio processing libraries: e.g. librosa, used for preprocessing audio data (noising, filtering).

[1277] Machine learning frameworks: e.g., TensorFlow, PyTorch, used for audio feature extraction and sentiment analysis.

[1278] Deep learning models: e.g. GAN (generative artificial network), StyleGAN, used to generate avatar images.

[1279] Speech recognition models, such as DeepSpeech, are used to extract features from speech data.

[1280] Data processing flow

[1281] The server first receives the voice data and performs preprocessing. Next, it extracts voice features and analyzes emotions. Based on this information, it selects an avatar model and generates an avatar image in real time. The generated image undergoes a quality check and is then sent to the user's smartphone. The smartphone displays the avatar image and receives user feedback.

[1282] Specific examples

[1283] Example 1: Real-time avatar generation during live streaming

[1284] When a user starts live streaming on their smartphone, they begin speaking into the voice input means. For example, they might say, "Hello, I'm so happy today!" This speech is sent to the server in real time. The server preprocesses the speech data and extracts features to analyze emotions. Based on the analysis results, an avatar model is selected and an avatar image is generated in real time. This avatar image is sent to the smartphone and visually displayed to the user.

[1285] Prompt Sentence Examples

[1286] "Hello, I'm so happy today!"

[1287] => Emotion: Joy

[1288] => Avatar Model: Smiling Anime Character

[1289] This detailed processing procedure generates a high-quality avatar that reflects the user's emotions in real time, enabling viewers to experience immersive live streaming.

[1290] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1291] Step 1:

[1292] The user begins speaking into the smartphone's microphone. The input is the user's voice data, for example, "Hello, I'm very happy today!" The output is recorded digital voice data. The device transmits this voice data to the server in real time.

[1293] Step 2:

[1294] The server receives the audio data sent from the terminal. The input is the digital audio data sent from the terminal. The server first preprocesses this audio data and performs noise removal. As a result, clean audio data with noise removed is obtained as output.

[1295] Step 3:

[1296] The server extracts features from the preprocessed speech data. The input is clean speech data with noise removed. Specifically, it uses FFT (Fast Fourier Transform) and cepstrum analysis to extract characteristic data such as formants, pitch, tone, and rhythm. The output is these speech feature data.

[1297] Step 4:

[1298] The server performs emotion analysis based on the speech feature data. The input is speech feature data. Neural networks or RNNs are used for emotion analysis to analyze emotions such as joy, sadness, anger, and surprise. The output is the analyzed emotion data.

[1299] Step 5:

[1300] The server selects the optimal avatar model based on the extracted speech features and analyzed emotion data. The input is speech feature data and emotion data. The server uses a machine learning algorithm to select the optimal avatar model from a pre-trained database. The output is the selected avatar model.

[1301] Step 6:

[1302] The server generates an avatar image based on the selected avatar model and the analyzed emotion data. The input is the avatar model and emotion data. The server uses a deep learning model (e.g., GAN, StyleGAN) to generate the avatar's face and expressions. The output is the generated avatar image.

[1303] Step 7:

[1304] The server checks the quality of the generated avatar image. The input is the generated avatar image. The server checks for facial distortion and unnatural expressions, and makes adjustments using post-processing algorithms if necessary. The output is the final quality-assured avatar image.

[1305] Step 8:

[1306] The server sends the final avatar image data to the user's smartphone. The input is a quality-guaranteed avatar image. The output is the avatar image displayed on the smartphone.

[1307] Step 9:

[1308] The device visually displays the received avatar image to the user. The input is the avatar image data sent from the server. The user can review the avatar image and provide feedback. The feedback is sent to the system through a robust user interface and used for further improvement.

[1309] Specific prompt examples:

[1310] "Hello, I'm so happy today!"

[1311] => Emotion: Joy

[1312] => Avatar Model: Smiling Anime Character

[1313] This processing step generates high-quality avatar images that reflect the user's emotions in real time, providing a more lifelike experience for the user and viewers.

[1314] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1315] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1316] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1317] [Fourth embodiment]

[1318] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1319] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1320] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1321] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1322] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1323] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1324] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1325] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1326] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1327] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1328] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1329] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1330] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1331] The present invention provides a system for generating avatars from voice data. The system aims to generate natural and realistic avatar images from a user's voice input, and is implemented by the following method and procedure.

[1332] System Overview

[1333] The system includes the following major components:

[1334] Voice input means

[1335] A means of transmitting audio data

[1336] A means of preprocessing audio data

[1337] A means of extracting speech features

[1338] A method for selecting avatar models based on speech features

[1339] A method for generating images based on avatar models

[1340] A way to check and adjust the quality of the generated avatars

[1341] A means for transmitting the generated avatar image data

[1342] A means of displaying an avatar image

[1343] A way to receive user feedback

[1344] Program processing

[1345] The program processing in this system will be explained in natural language below.

[1346] 1. Audio input

[1347] The user speaks into the device, and can include the message they want to convey and the characteristics of the avatar they want to create. For example, they might say, "Hello, my name is Sato. Please create an anime character for me."

[1348] 2. Sending audio data

[1349] The terminal records the user's voice and transmits the digital voice data to the server.

[1350] 3. Preprocessing of audio data

[1351] The server receives the audio data and first performs noise reduction, which filters out background noise and unwanted sounds.

[1352] The audio data is then converted into a digital format that makes it easier to analyze.

[1353] 4. Extraction of speech features

[1354] The server extracts speech features from the speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[1355] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[1356] 5. Select an avatar model

[1357] The server selects the most suitable avatar model based on the extracted voice features. For example, if the voice is bright and high-pitched, it selects a youthful anime character.

[1358] To select a model, a machine learning algorithm is used, referencing pre-trained voice-character pair data.

[1359] 6. Image Generation

[1360] The server generates an image based on the selected avatar model, specifically using deep learning models such as GAN (Generative Adversarial Network) and StyleGAN.

[1361] This generates a facial pattern corresponding to the audio features.

[1362] 7. Avatar quality check and adjustment

[1363] The server checks the quality of the generated avatar, checking for unnatural features such as facial distortion or unnatural facial expressions.

[1364] If necessary, fine-tune using post-processing algorithms.

[1365] 8. Sending and displaying avatar images

[1366] The server generates the final avatar image data and sends it to the user's terminal.

[1367] The device displays the received avatar image to the user, who can then review the avatar image and provide feedback if necessary.

[1368] Specific examples

[1369] Below are some specific usage examples.

[1370] Use case 1: Anime-style avatar generation

[1371] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create an anime character for me."

[1372] 2. The device records the audio and sends it to the server.

[1373] 3. The server preprocesses the audio data, removes noise, and extracts features.

[1374] 4. The server selects a youthful anime character model based on the features.

[1375] 5. The server uses GAN to generate an animated avatar.

[1376] 6. The server checks the quality of the generated avatar and makes adjustments if necessary.

[1377] 7. The server sends the avatar image to the device.

[1378] 8. The device displays the avatar to the user, who confirms it.

[1379] This system allows users to quickly generate a natural voice-based avatar to enhance visual communication.

[1380] The processing flow will be explained below.

[1381] Step 1:

[1382] The user speaks into the device's microphone, describing the desired avatar's characteristics and a message. For example, they might say, "Hello, I'm Sato. Please create an anime character for me."

[1383] Step 2:

[1384] The device records the user's voice and generates digital audio data that is stored in a common audio file format (e.g., WAV, MP3).

[1385] Step 3:

[1386] The device sends the recorded audio data to a server, using an internet connection for data transfer.

[1387] Step 4:

[1388] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[1389] Step 5:

[1390] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are extracted using FFT (Fast Fourier Transform) and cepstrum analysis.

[1391] Step 6:

[1392] The server selects an appropriate avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest, SVM) to select the best model from a pre-trained voice-character pair database.

[1393] Step 7:

[1394] The server generates an avatar image based on the selected avatar model, and uses a deep learning model (e.g., GAN, StyleGAN) to generate a facial pattern corresponding to the audio features.

[1395] Step 8:

[1396] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[1397] Step 9:

[1398] The server sends the final avatar image data to the device, where it is saved in high-resolution format and provided to the user.

[1399] Step 10:

[1400] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[1401] This series of steps allows users to quickly generate a natural, high-quality avatar based on their voice.

[1402] Example 1

[1403] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1404] Conventional avatar generation systems have had many problems when generating natural and realistic avatars from user voice, particularly in the preprocessing of voice data, feature extraction, and final image generation and quality adjustment. Specifically, the efficiency of noise removal from voice data was low, and the accuracy of extracted features was insufficient, making it often impossible to select the optimal avatar model. Furthermore, the quality of the generated images was unstable, with many unnatural parts. A method to resolve these issues is needed.

[1405] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1406] In this invention, the server includes means for performing FFT analysis on the voice data, means for generating avatar images using GAN, and means for adjusting the quality of the generated images using a post-processing algorithm, which enables highly accurate analysis and noise removal of the voice data, selection of the optimal avatar model, and generation and adjustment of high-quality avatar images.

[1407] "Voice input means" refers to a device or interface that allows a user to input voice, such as a microphone or voice recognition software.

[1408] "Means for transmitting audio data" refers to the communications means for transmitting audio data from the user's device to the server, typically including an internet connection and the HTTP protocol.

[1409] "Means for pre-processing audio data" refers to processes for converting recorded audio data into an analyzable format, including, for example, noise reduction and digital conversion.

[1410] "Means for extracting speech features" refers to techniques for analyzing and extracting necessary speech features from preprocessed speech data, including FFT (Fast Fourier Transform) and cepstrum analysis.

[1411] "Means for selecting an avatar model based on speech features" refers to an algorithm for selecting an optimal avatar model based on extracted speech features, including machine learning algorithms.

[1412] "Means for generating images based on an avatar model" refers to technology for generating the final avatar image based on the selected avatar model, including deep learning models such as Generative Adversarial Networks (GANs) and StyleGANs.

[1413] "Means for checking and adjusting the quality of the generated avatar" refers to technology, including post-processing algorithms, for checking whether the generated avatar image is appropriate and correcting it if necessary.

[1414] "Means for transmitting generated avatar image data" refers to a communication means for transmitting the generated avatar image from the server to the user's device, such as the HTTP protocol.

[1415] "Means for displaying an avatar image" refers to technology for displaying the received avatar image on a user's device, including a display or monitor.

[1416] "Means for receiving user feedback" refers to an interface for collecting user opinions and comments about the generated avatar, including a GUI or web form.

[1417] "FFT analysis" refers to the fast Fourier transform technology used to efficiently analyze the frequency components of audio data.

[1418] "Means for generating avatar images using GAN" refers to technology for automatically generating natural-looking avatar images using a generative artificial network (GAN).

[1419] "Post-processing algorithms" refers to post-processing techniques that apply additional adjustments and modifications to the generated images, including quality improvement using OpenCV.

[1420] MODE FOR CARRYING OUT THE INVENTION

[1421] The present invention is a system that generates realistic and natural avatar images based on voice input. The series of processes that generate an avatar from a user's voice input and display that image on a terminal will be described below.

[1422] First, the user speaks into the device. This voice input is collected using a microphone and voice recognition software. For example, the user might give a voice command such as, "Hello, I'm Suzuki. Please create an anime character for me." This voice data is then digitally recorded on the device.

[1423] The device then sends the recorded audio data to the server using an HTTP POST request over the internet connection, in a common audio file format such as WAV.

[1424] The server performs preprocessing on the received audio data. This preprocessing includes noise reduction, which uses filters to remove background noise and unwanted sounds to clean the audio data. It also analyzes the frequency components of the audio data using FFT (Fast Fourier Transform). This allows it to extract audio features such as pitch (high and low pitch), formants (resonant frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[1425] Based on the extracted features, the server selects the optimal avatar model. This selection is performed using a machine learning algorithm, such as a random forest or deep neural network, to match the voice features with the avatar model. If the user's voice has a youthful tone, a youthful anime character model will be selected.

[1426] After the model is selected, the server generates an avatar image using a generative AI model such as a Generative Adversarial Network (GAN) or StyleGAN. This process generates a natural-looking avatar that reflects the audio features. The generated avatar image is then quality checked and adjusted using post-processing algorithms. For example, libraries such as OpenCV are used to correct unnatural facial distortions and imperfections in facial expressions.

[1427] The final avatar image is sent from the server to the device. Again, the HTTP protocol is generally used. The device displays the received avatar image to the user. The user can check the displayed avatar and provide feedback if necessary. This feedback is reflected in the next avatar generation, enabling the generation of a higher quality avatar.

[1428] Specific examples

[1429] Use case 1: Anime-style avatar generation

[1430] Example prompt sentence:

[1431] "Hello, I'm Suzuki. Please make me an anime character."

[1432] The system allows users to quickly generate natural voice-based avatars to enhance visual communication, a process achieved through the use of highly accurate voice analysis and generative AI models.

[1433] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1434] Program processing flow

[1435] Step 1:

[1436] The user speaks to the device, for example, "Hello, my name is Sato. Please create an anime character for me." At this time, the voice input is recorded as digital voice data via a microphone.

[1437] Input: User's voice

[1438] Output: Digital audio data (WAV, etc.)

[1439] Step 2:

[1440] The device sends the recorded audio data in digital format to the server using an HTTP POST request.

[1441] Input: Digital audio data

[1442] Output: Audio data sent in HTTP POST request

[1443] Step 3:

[1444] The server first denoises the received audio data to convert it into an analyzable format, using a filter to remove background noise and obtain clean audio data, then resamples the audio data to a standard sampling rate.

[1445] Input: Transmitted audio data

[1446] Output: Preprocessed, clean audio data

[1447] Step 4:

[1448] The server analyzes the frequency components from the preprocessed audio data using FFT (Fast Fourier Transform), extracting audio features such as pitch, formants, tone, and rhythm. The features are stored in list or vector format.

[1449] Input: Preprocessed clean audio data

[1450] Output: A list or vector of audio features

[1451] Step 5:

[1452] The server selects the optimal avatar model based on the extracted voice features. It uses a machine learning algorithm (e.g., random forest or deep neural network) and references a database of voice features and avatar models. The selected model is recorded as the selection result.

[1453] Input: List of audio features

[1454] Output: Selected avatar model

[1455] Step 6:

[1456] The server generates an avatar image based on the selected avatar model using a generative AI model (e.g., GAN or StyleGAN). The generation process uses multiple neural networks to generate the final avatar image.

[1457] Input: Selected avatar model

[1458] Output: Generated avatar image

[1459] Step 7:

[1460] The server performs a quality check on the generated avatar images, detecting any unnatural features (facial distortions or imperfections in expressions) and correcting them using post-processing algorithms (e.g., OpenCV).

[1461] Input: Generated avatar image

[1462] Output: Quality adjusted avatar image

[1463] Step 8:

[1464] The server then sends the final avatar image data to the user's device, again using the HTTP protocol.

[1465] Input: Quality adjusted avatar image

[1466] Output: Avatar image data sent in the HTTP response

[1467] Step 9:

[1468] The device displays the received avatar image to the user. The avatar image can be visually confirmed using the display. The user can check the displayed avatar image and provide feedback through the device.

[1469] Input: Avatar image data received in the HTTP response

[1470] Output: Avatar image displayed on the screen

[1471] Through this series of steps, users can quickly generate realistic and natural avatar images based on voice input, enhancing visual communication.

[1472] (Application example 1)

[1473] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1474] In conventional virtual stores, there has been no system that provides an assistant avatar that can interact with users in real time. This has led to the issue of not being able to improve the interactive shopping experience in stores. There has also been a lack of systems that can generate and display an appropriate avatar based on the user's voice request.

[1475] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1476] In this invention, the server includes a voice input means, a voice data transmission means, a voice data preprocessing means, a voice feature extraction means, a voice feature selection means, an avatar model selection means based on the voice feature, an image generation means based on the avatar model, a quality check and adjustment means for the generated avatar, a transmission means for the generated avatar image data, a display means for the avatar image, a user feedback receiving means, and a generation means for an assistant avatar to provide an interactive shopping experience in a virtual store. This allows a user to easily generate an avatar through voice input and realize an interactive shopping experience.

[1477] "Voice input means" refers to a device for capturing the user's voice, and includes a microphone and the voice input function of a smartphone.

[1478] "Means for transmitting audio data" refers to a communications interface or network protocol for transmitting recorded audio data to a server or other processing system.

[1479] "Audio data preprocessing means" refers to algorithms or software that remove noise from recorded audio data and convert the data into a format that is easier to analyze.

[1480] "Means for extracting speech features" refers to technology for extracting characteristic elements of speech, such as pitch, formants, tone, and rhythm, from speech data.

[1481] The "means for selecting an avatar model based on speech features" refers to a machine learning algorithm or database for selecting the optimal avatar model based on the extracted speech features.

[1482] "Means for generating images based on an avatar model" refers to technologies for generating actual avatar images from a selected avatar model, and includes deep learning technologies such as GAN (generative adversarial networks) and StyleGAN.

[1483] "Means for checking and adjusting the quality of the generated avatar" refers to algorithms or software for evaluating the quality of the generated avatar image and correcting any unnatural parts.

[1484] The "means for transmitting generated avatar image data" refers to a communication interface or network protocol for transmitting the avatar image data generated by the server to a user's device.

[1485] The "means for displaying an avatar image" refers to a display device or application that a user uses to view the generated avatar image.

[1486] "Means for receiving user feedback" refers to an interface or system for collecting evaluations and opinions from users.

[1487] "Means for generating assistant avatars to provide an interactive shopping experience within a virtual store" refers to a technology that generates an avatar based on voice data to interact with users in a virtual store environment and provide guidance and support.

[1488] A system for implementing the present invention includes a voice input means, a means for transmitting voice data, a means for preprocessing the voice data, a means for extracting voice features, a means for selecting an avatar model based on the voice features, a means for generating an image based on the avatar model, a means for checking and adjusting the quality of the generated avatar, a means for transmitting the generated avatar image data, a means for displaying the avatar image, and a means for receiving user feedback.

[1489] First, the user uses a voice input means such as a smartphone or a head-mounted display to input a voice such as "Hello, I'm looking for shoes." The voice data is recorded through the microphone of the device and transmitted to the server via the network by a means for transmitting voice data.

[1490] The server preprocesses the received audio data, removing noise and converting it to digital form. Specifically, it uses the Python speech_recognition library. From the preprocessed audio data, the server extracts audio features. Using FFT (Fast Fourier Transform) and cepstrum analysis, audio features such as pitch, formant, tone, and rhythm are extracted.

[1491] Based on the extracted voice features, the server selects the optimal avatar model using machine learning algorithms and pre-trained voice-character pair data. For example, if the voice tone is bright and high, a youthful anime character model will be selected at this stage.

[1492] Based on the selected avatar model, the server generates an avatar image using deep learning techniques (e.g., GAN or StyleGAN). The generated avatar image is then quality-evaluated by the server to ensure there are no unnatural features. If there are any defects, they are corrected using a post-processing algorithm.

[1493] The final avatar image data after adjustment is sent from the server to the device. The device displays the received avatar image on the user's display device. The user can review the avatar and provide feedback if necessary. This feedback is sent to the server and is used to improve the system and increase the accuracy of avatar generation.

[1494] As a specific example of use, consider a situation where a user speaks to a device, "Hello, I'm Suzuki. Please create an anime character for me." The server selects an avatar model from this voice data, generates an anime-style avatar using GAN, and finally displays it on the device as a series of processes.

[1495] Another example of a prompt when generating an avatar using a generative AI model is, "Please generate an avatar that is a youthful female character with a friendly smile."

[1496] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1497] Step 1:

[1498] The user inputs voice into the device. This voice input includes the characteristics and wishes of the avatar the user wants to create. For example, the user might say, "Hello, my name is Sato. Please create an anime character for me." The voice data is recorded via a smartphone or microphone. The input is voice data, and the output is a recorded voice file.

[1499] Step 2:

[1500] The device sends the recorded audio data to the server, using communication methods such as the Internet or Bluetooth. The input is an audio file, and the output is the transmission of audio data to the server.

[1501] Step 3:

[1502] The server preprocesses the received audio data. Specifically, it performs noise removal and digital conversion of the audio signal. This preprocessing uses the speech_recognition library and other audio processing algorithms. The input is audio data, and the output is the preprocessed audio data.

[1503] Step 4:

[1504] The server extracts features from the speech data. For example, it uses FFT (Fast Fourier Transform) and cepstrum analysis techniques to extract speech features such as pitch, formant, tone, and rhythm. The input is preprocessed speech data, and the output is speech features.

[1505] Step 5:

[1506] The server selects an avatar model based on the extracted voice features. Here, a machine learning algorithm is used to compare the extracted voice features with pre-trained data and select the optimal avatar model. The input is the voice features, and the output is the selected avatar model.

[1507] Step 6:

[1508] The server generates an image based on the selected avatar model. Specifically, it uses a generative AI model such as GAN or StyleGAN to generate the avatar image. It uses a prompt such as "Please generate an avatar that is a youthful female character with a friendly smile." The input is the avatar model, and the output is the generated avatar image data.

[1509] Step 7:

[1510] The server checks the quality of the generated avatar and adjusts any unnatural parts. Quality assessment algorithms and post-processing algorithms are used here. The input is the generated avatar image data, and the output is the adjusted avatar image data.

[1511] Step 8:

[1512] The server sends the adjusted avatar image data to the user's device. Data is transferred using a network protocol. The input is the adjusted avatar image data, and the output is data sent to the device.

[1513] Step 9:

[1514] The device displays the received avatar image. Specifically, it is displayed on a smartphone display or a head-mounted display. The input is avatar image data, and the output is the displayed avatar image.

[1515] Step 10:

[1516] The user checks the generated avatar image and sends any feedback they wish to provide to the server via their device. The input is the user's feedback, and the output is the feedback data sent to the server.

[1517] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1518] The present invention aims to generate natural and realistic avatar images that reflect the user's emotions by combining an emotion engine with a system that generates avatars from voice data. This system is implemented using the following components and procedures.

[1519] System Overview

[1520] The system includes the following major components:

[1521] Voice input means

[1522] A means of transmitting audio data

[1523] A means of preprocessing audio data

[1524] A means of extracting speech features

[1525] Emotion Engine

[1526] A method for selecting avatar models based on speech features and emotions

[1527] Avatar model and means for generating images based on emotions

[1528] A way to check and adjust the quality of the generated avatars

[1529] A means for transmitting the generated avatar image data

[1530] A means of displaying an avatar image

[1531] A way to receive user feedback

[1532] Program processing

[1533] The program processing in this system will be explained in natural language below.

[1534] 1. Audio input

[1535] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Sato. Please create a lively anime character."

[1536] 2. Sending audio data

[1537] The device records the user's voice and generates digital audio data, which is saved in a common audio file format (e.g., WAV, MP3) and sent to a server.

[1538] 3. Preprocessing of audio data

[1539] The server receives the audio data and performs pre-processing: first, noise reduction is performed to filter out background noise and unwanted sounds.

[1540] The audio data is then converted into a digital format that makes it easier to analyze.

[1541] 4. Extraction of speech features

[1542] The server extracts features from the preprocessed speech data, including pitch (high / low), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm.

[1543] These features are extracted using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[1544] 5. Emotional Recognition

[1545] The emotion engine recognizes the user's emotions from the voice data. The emotion engine analyzes voice parameters and distinguishes emotions such as joy, sadness, anger, and surprise.

[1546] Machine learning algorithms (e.g., neural networks, RNNs) are used to identify emotions from a pre-trained dataset.

[1547] 6. Select an avatar model

[1548] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character.

[1549] To select a model, a machine learning algorithm is used to select the optimal model from a pre-trained database of voice-character pairs.

[1550] 7. Image Generation

[1551] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[1552] 8. Avatar quality check and adjustment

[1553] The server checks the quality of the generated avatar, specifically for facial distortions and unnatural expressions, and, if necessary, fine-tunes it using post-processing algorithms.

[1554] 9. Sending and displaying avatar images

[1555] The server generates the final avatar image data and sends it to the user's device, where it is saved in high-resolution format and provided to the user.

[1556] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything about it they don't like. For example, they can provide specific feedback such as, "I wish the facial expression was more smiling."

[1557] Specific examples

[1558] Use case 1: Generating anime-style avatars that reflect emotions

[1559] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[1560] 2. The device records the audio and sends it to the server.

[1561] 3. The server preprocesses the audio data, removes noise, and extracts features.

[1562] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[1563] 5. The server selects a lively anime character model based on the features and emotions.

[1564] 6. The server uses GAN to generate an animated avatar with a happy expression.

[1565] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[1566] 8. The server sends the avatar image to the device.

[1567] 9. The device displays the avatar to the user, who confirms it.

[1568] The system allows users to quickly generate natural, high-quality, and emotionally sensitive avatars based on their voice.

[1569] The processing flow will be explained below.

[1570] Step 1:

[1571] The user speaks into the device's microphone, describing the desired avatar's characteristics, message, and emotions. For example, they might say, "Hello, I'm Suzuki. Please create a lively anime character."

[1572] Step 2:

[1573] The device records the user's voice and generates digital audio data that is stored in common audio file formats such as WAV or MP3.

[1574] Step 3:

[1575] The device sends the recorded audio data to the server, using an internet connection to transfer the data in a reliable manner (e.g. HTTPS).

[1576] Step 4:

[1577] The server receives the audio data and first performs noise reduction, using digital signal processing techniques (e.g. bandpass filters) to filter out background noise and unwanted sounds.

[1578] Step 5:

[1579] The server extracts features from the preprocessed speech data. These features include pitch (high / low pitch), formants (frequency bands estimated from the shape of the mouth), tone (timbre), and rhythm. These features are analyzed using techniques such as FFT (Fast Fourier Transform) and cepstrum analysis.

[1580] Step 6:

[1581] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes voice parameters to identify emotions such as joy, sadness, anger, and surprise. Specifically, it uses a machine learning algorithm (e.g., neural network, RNN) to identify emotions from a pre-trained dataset.

[1582] Step 7:

[1583] The server selects the optimal avatar model based on the voice features and the output from the emotion engine. For example, if the voice is bright and high-pitched and the emotion is "joy," it selects a lively anime character. The model selection uses a machine learning algorithm and references a pre-trained voice-character pair database.

[1584] Step 8:

[1585] The server generates an avatar image based on the selected avatar model and the recognized emotion. It uses a deep learning model (e.g., GAN, StyleGAN) to generate facial patterns and expressions corresponding to the voice features and emotions.

[1586] Step 9:

[1587] The server checks the quality of the generated avatar, specifically checking for facial distortions and unnatural expressions, and automatically corrects any unnatural features using post-processing algorithms.

[1588] Step 10:

[1589] The server sends the final avatar image data to the device, where it is stored in high-resolution format and may be compressed depending on the requirements of the service being used.

[1590] Step 11:

[1591] The device displays the received avatar image to the user, who can then review it and provide feedback if there is anything they don't like about it. For example, the user can make a specific request, such as "I want the facial expression to be more smiling."

[1592] Specific examples

[1593] Use case 1: Generating anime-style avatars that reflect emotions

[1594] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[1595] 2. The device records the audio and sends it to the server.

[1596] 3. The server preprocesses the audio data, removes noise, and extracts features.

[1597] 4. The emotion engine recognizes the user's emotion "joy" from the voice data.

[1598] 5. The server selects a lively anime character model based on the features and emotions.

[1599] 6. The server uses GAN to generate an animated avatar with a happy expression.

[1600] 7. The server checks the quality of the generated avatar and makes adjustments if necessary.

[1601] 8. The server sends the avatar image to the device.

[1602] 9. The device displays the avatar to the user, who confirms it.

[1603] Example 2

[1604] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1605] Conventional avatar generation systems have had difficulty generating natural and realistic avatar images that reflect the user's emotions. A particular technical challenge has been properly recognizing emotions from voice data and creating an avatar with corresponding facial expressions. Therefore, a system that can easily generate avatars that accurately reflect the user's intentions and emotions has been sought.

[1606] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1607] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, means for recognizing emotions using an emotion engine, means for selecting an avatar model based on the voice features and emotions, means for generating an image based on the avatar model and emotions, means for checking and adjusting the quality of the generated avatar, and means for transmitting the generated avatar image data. This makes it possible to recognize a user's emotions from voice data and quickly generate natural and realistic avatar images that reflect those emotions.

[1608] The "voice input means" is a means that enables a user to utter voice data into the microphone of the terminal.

[1609] The "means for transmitting audio data" refers to a means for transferring recorded audio data to a server in digital format.

[1610] "Means for preprocessing audio data" refers to means for removing noise and filtering audio data, and converting the audio data into a format that is easy to analyze.

[1611] The "means for extracting speech features" refers to a means for extracting features such as pitch, formant, tone, and rhythm from speech data.

[1612] The "emotion engine" is an engine that uses machine learning algorithms to analyze the user's emotions from voice data and identify emotions such as joy, sadness, anger, and surprise.

[1613] The "means for selecting an avatar model" is a means for selecting an optimal avatar model based on speech features and emotions.

[1614] The "means for generating an image" is a means using a deep learning model to generate an avatar image based on a selected avatar model and a recognized emotion.

[1615] The "means for checking and adjusting the quality of the generated avatar" refers to a means for checking the quality of the generated avatar image and adjusting facial distortions and unnatural expressions using a post-processing algorithm.

[1616] The "means for transmitting generated avatar image data" refers to a means for transmitting the final avatar image data to a user's device.

[1617] The "means for displaying an avatar image" refers to a means for displaying the received avatar image on the user's terminal.

[1618] The "means for receiving user feedback" is a means for a user to provide opinions or requests for corrections to the generated avatar image.

[1619] The present invention is a system that recognizes a user's emotions from voice data and generates a natural and realistic avatar image that reflects those emotions. This system is composed of multiple hardware and software components. Specifically, it is implemented using the following components and procedures.

[1620] Main components

[1621] Voice input method: The user speaks into the device's microphone, describing the desired avatar's characteristics, messages, and emotions. The device captures the voice using a microphone sound input library (e.g., PyAudio).

[1622] How audio data is transmitted: The device saves the recorded audio data in a common audio file format (e.g., WAV, MP3) and transmits it to the server using an HTTP request.

[1623] A means of pre-processing the audio data: The server uses an audio processing library (e.g., Librosa) to denoise and filter the audio data, then convert it into a digital format that is easier to analyze.

[1624] Method for extracting speech features: The server uses FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formant, tone, and rhythm from the speech data.

[1625] Emotion engine: The server uses an emotion engine (e.g., neural network or RNN) to analyze the user's emotions from the voice features and identify emotions such as joy, sadness, anger, and surprise.

[1626] Method for selecting an avatar model: The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on audio features and emotion information.

[1627] Means for generating images: The server uses GAN (generative adversarial network) or StyleGAN to generate avatar images based on the selected avatar model and recognized emotions.

[1628] A means to check and adjust the quality of the generated avatar: The server checks the generated avatar image for facial distortion and unnatural expressions, and performs post-processing using image processing algorithms (e.g., OpenCV) to make any necessary fine-tuning.

[1629] Means for sending the generated avatar image data: The server saves the final avatar image data in a high-resolution format (e.g., PNG) and sends it to the user's device.

[1630] A method for displaying the avatar image: The device displays the received avatar image. For example, HTML5 Use tags to display in the browser.

[1631] A means of receiving user feedback: Users can provide specific opinions and requests for corrections to the displayed avatar image. For example, they can send feedback via their device, such as "I want the facial expression to be more smiling."

[1632] Usage example

[1633] Specific examples are shown below.

[1634] Use case 1: Generating anime-style avatars that reflect emotions

[1635] 1. The user speaks to the terminal, "Hello, I'm Suzuki. Please create a lively anime character."

[1636] 2. The device records the audio and sends it to the server in WAV format.

[1637] 3. The server uses Librosa to denoise and filter the audio data, making it easier to analyze.

[1638] 4. The server extracts features from the audio data using FFT and cepstrum analysis.

[1639] 5. The emotion engine identifies the user's emotion "joy" from the voice.

[1640] 6. The server uses random forests to select a model of a lively anime character based on features and emotions.

[1641] 7. The server uses GAN to generate an animated avatar with a happy expression.

[1642] 8. The server checks the quality of the generated avatar using OpenCV and makes adjustments if necessary.

[1643] 9. The server sends the final avatar image to the device.

[1644] 10. The device displays the received image and the user confirms it.

[1645] Prompt Sentence Examples

[1646] Generate a new avatar image using the voice data of a user saying "Hello, I'm Suzuki. Please create a lively anime character." The emotion to use is "joy."

[1647] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1648] Step 1:

[1649] The user speaks into the device's microphone, including the desired avatar's characteristics, message, and emotion. For example, "Hello, I'm Sato. Please create a lively anime character." This voice is input as an analog signal.

[1650] Step 2:

[1651] The device uses a microphone audio input library (e.g., PyAudio) to capture the user's voice and save it in a digital format (WAV or MP3). The saved voice data is then output.

[1652] Step 3:

[1653] The device sends audio data to the server via an HTTP request (e.g., using Python's requests library). This operation inputs digital audio data and transfers it to the server.

[1654] Step 4:

[1655] The server denoises the audio data using an audio processing library such as Librosa, filtering out background noise and unwanted sounds. The input is digital audio data, and the output is clean audio data after noise reduction.

[1656] Step 5:

[1657] The server uses an audio processing library to convert the clean audio data into a more analyzable form. This calculation includes formatting in digital form. The converted audio data is output.

[1658] Step 6:

[1659] The server performs FFT (Fast Fourier Transform) and cepstrum analysis to extract speech features such as pitch, formants, tone, and rhythm. The input is preprocessed speech data, and the extracted speech features are output.

[1660] Step 7:

[1661] The emotion engine analyzes the user's emotions using a machine learning algorithm (e.g., neural network, RNN) based on the extracted voice features. The input is the voice features, and the output is the identified emotion (e.g., joy, sadness).

[1662] Step 8:

[1663] The server selects the optimal avatar model using a machine learning algorithm (e.g., random forest) based on the identified emotions and voice features. The input is the voice features and emotion information, and the selected avatar model is output.

[1664] Step 9:

[1665] The server generates an avatar image using a deep learning model (e.g., GAN, StyleGAN) based on the selected avatar model and the identified emotion. The input is the avatar model and emotion information, and the generated avatar image is output.

[1666] Step 10:

[1667] The server uses image processing algorithms (e.g., OpenCV) to check the quality of the generated avatar image and check for facial distortions and unnatural expressions. If necessary, post-processing is performed and a fine-tuned avatar image is output.

[1668] Step 11:

[1669] The server saves the final avatar image in a high-resolution format (e.g., PNG) and sends it to the user's device as an HTTP response. The input is the adjusted avatar image data, and the output is the sent image data.

[1670] Step 12:

[1671] The device receives an avatar image and sends it to HTML5. It displays the image to the user using tags, etc. The user can review the image and provide specific feedback (e.g., "Please smile more") if there is anything they don't like. The input is avatar image data, and the displayed avatar image is output.

[1672] (Application example 2)

[1673] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1674] Conventional avatar generation systems have had difficulty generating avatars that accurately reflect a user's emotions in real time. In particular, live streaming and real-time content distribution require immediate reflection of the user's emotions and voice characteristics. To solve this problem, a system is needed that performs highly accurate emotion analysis and voice feature extraction, selects an avatar model based on the results, and generates images in real time.

[1675] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1676] In this invention, the server includes means for preprocessing voice data, means for extracting voice features, and means for analyzing emotions, which enable real-time analysis of emotions from a user's voice, selection of an avatar model based on the emotions, and generation of an image, thereby enabling the generation of avatars that accurately reflect the user's emotions in live streaming or real-time content distribution.

[1677] The "voice input means" is a device or mechanism for acquiring the user's voice as digital voice data.

[1678] The "means for transmitting voice data" is a mechanism for transmitting the acquired voice data to another device or server.

[1679] "Means for preprocessing audio data" refers to a processing device or method for removing noise from audio data and making it easier to analyze.

[1680] A "means for extracting audio features" is a device or method for extracting characteristic data such as formants, pitch, tone, and rhythm from audio data.

[1681] "Means for analyzing emotions" refers to devices or methods for identifying and determining a user's emotions from voice data.

[1682] The "means for selecting an avatar model based on speech features and emotions" refers to a device or method for selecting an optimal avatar model based on the acquired speech features and analyzed emotions.

[1683] The "means for generating an image based on an avatar model and an emotion" refers to a device or method for generating a specific avatar image according to a selected avatar model and an analyzed emotion.

[1684] "Means for checking and adjusting the quality of the generated avatar" refers to a device or method for inspecting the generated avatar image for unnatural features or defects and correcting them if necessary.

[1685] The "means for transmitting the generated avatar image data" is a mechanism for transmitting the generated avatar image data to a user's terminal or another system.

[1686] The "means for displaying an avatar image" refers to a display device or a display program for visually displaying the generated avatar image.

[1687] The "means for receiving user feedback" refers to a device or interface for obtaining user opinions and impressions regarding the generated avatar image.

[1688] This invention relates to an avatar generation system that combines voice data and emotion analysis. The purpose of this system is to generate and display avatars in real time that reflect the user's emotions in real-time live streaming and content distribution.

[1689] Main system components

[1690] The system requires the following major components:

[1691] Voice input means

[1692] A means of transmitting audio data

[1693] A means of preprocessing audio data

[1694] A means of extracting speech features

[1695] A means of analyzing emotions

[1696] A method for selecting avatar models based on speech features and emotions

[1697] Avatar model and means for generating images based on emotions

[1698] A way to check and adjust the quality of the generated avatars

[1699] A means for transmitting the generated avatar image data

[1700] A means of displaying an avatar image

[1701] A way to receive user feedback

[1702] Hardware and software used

[1703] This system requires high-performance hardware and advanced software. The main hardware and software used are described below.

[1704] Hardware

[1705] Smartphone: An iOS or Android device used to input voice and display results.

[1706] High-performance server: Equipped with a GPU, it performs data preprocessing, voice analysis, emotion analysis, and avatar generation.

[1707] software

[1708] Audio processing libraries: e.g. librosa, used for preprocessing audio data (noising, filtering).

[1709] Machine learning frameworks: e.g., TensorFlow, PyTorch, used for audio feature extraction and sentiment analysis.

[1710] Deep learning models: e.g. GAN (generative artificial network), StyleGAN, used to generate avatar images.

[1711] Speech recognition models, such as DeepSpeech, are used to extract features from speech data.

[1712] Data processing flow

[1713] The server first receives the voice data and performs preprocessing. Next, it extracts voice features and analyzes emotions. Based on this information, it selects an avatar model and generates an avatar image in real time. The generated image undergoes a quality check and is then sent to the user's smartphone. The smartphone displays the avatar image and receives user feedback.

[1714] Specific examples

[1715] Example 1: Real-time avatar generation during live streaming

[1716] When a user starts live streaming on their smartphone, they begin speaking into the voice input means. For example, they might say, "Hello, I'm so happy today!" This speech is sent to the server in real time. The server preprocesses the speech data and extracts features to analyze emotions. Based on the analysis results, an avatar model is selected and an avatar image is generated in real time. This avatar image is sent to the smartphone and visually displayed to the user.

[1717] Prompt Sentence Examples

[1718] "Hello, I'm so happy today!"

[1719] => Emotion: Joy

[1720] => Avatar Model: Smiling Anime Character

[1721] This detailed processing procedure generates a high-quality avatar that reflects the user's emotions in real time, enabling viewers to experience immersive live streaming.

[1722] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1723] Step 1:

[1724] The user begins speaking into the smartphone's microphone. The input is the user's voice data, for example, "Hello, I'm very happy today!" The output is recorded digital voice data. The device transmits this voice data to the server in real time.

[1725] Step 2:

[1726] The server receives the audio data sent from the terminal. The input is the digital audio data sent from the terminal. The server first preprocesses this audio data and performs noise removal. As a result, clean audio data with noise removed is obtained as output.

[1727] Step 3:

[1728] The server extracts features from the preprocessed speech data. The input is clean speech data with noise removed. Specifically, it uses FFT (Fast Fourier Transform) and cepstrum analysis to extract characteristic data such as formants, pitch, tone, and rhythm. The output is these speech feature data.

[1729] Step 4:

[1730] The server performs emotion analysis based on the speech feature data. The input is speech feature data. Neural networks or RNNs are used for emotion analysis to analyze emotions such as joy, sadness, anger, and surprise. The output is the analyzed emotion data.

[1731] Step 5:

[1732] The server selects the optimal avatar model based on the extracted speech features and analyzed emotion data. The input is speech feature data and emotion data. The server uses a machine learning algorithm to select the optimal avatar model from a pre-trained database. The output is the selected avatar model.

[1733] Step 6:

[1734] The server generates an avatar image based on the selected avatar model and the analyzed emotion data. The input is the avatar model and emotion data. The server uses a deep learning model (e.g., GAN, StyleGAN) to generate the avatar's face and expressions. The output is the generated avatar image.

[1735] Step 7:

[1736] The server checks the quality of the generated avatar image. The input is the generated avatar image. The server checks for facial distortion and unnatural expressions, and makes adjustments using post-processing algorithms if necessary. The output is the final quality-assured avatar image.

[1737] Step 8:

[1738] The server sends the final avatar image data to the user's smartphone. The input is a quality-guaranteed avatar image. The output is the avatar image displayed on the smartphone.

[1739] Step 9:

[1740] The device visually displays the received avatar image to the user. The input is the avatar image data sent from the server. The user can review the avatar image and provide feedback. The feedback is sent to the system through a robust user interface and used for further improvement.

[1741] Specific prompt examples:

[1742] "Hello, I'm so happy today!"

[1743] => Emotion: Joy

[1744] => Avatar Model: Smiling Anime Character

[1745] This processing step generates high-quality avatar images that reflect the user's emotions in real time, providing a more lifelike experience for the user and viewers.

[1746] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1747] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1748] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1749] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1750] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1751] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1752] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1753] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1754] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1755] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1756] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1757] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1758] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1759] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1760] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1761] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1762] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1763] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1764] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1765] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1766] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1767] The following is further disclosed regarding the above embodiment.

[1768] (Claim 1)

[1769] A voice input means;

[1770] means for transmitting audio data;

[1771] means for preprocessing the audio data;

[1772] means for extracting speech features;

[1773] means for selecting an avatar model based on speech features;

[1774] A means for generating an image based on an avatar model;

[1775] A means to check and adjust the quality of the generated avatar;

[1776] means for transmitting the generated avatar image data;

[1777] means for displaying an avatar image;

[1778] A way to receive user feedback

[1779] A system including:

[1780] (Claim 2)

[1781] 2. The system of claim 1, wherein the speech features include formants and pitch.

[1782] (Claim 3)

[1783] 10. The system of claim 1, further comprising means for generating an image using a deep learning model.

[1784] "Example 1"

[1785] (Claim 1)

[1786] A voice input means;

[1787] means for transmitting audio data;

[1788] means for preprocessing the audio data;

[1789] means for extracting speech features;

[1790] means for selecting an avatar model based on speech features;

[1791] A means for generating an image based on an avatar model;

[1792] A means to check and adjust the quality of the generated avatar;

[1793] means for transmitting the generated avatar image data;

[1794] means for displaying an avatar image;

[1795] a means for receiving user feedback;

[1796] means for performing FFT analysis on the audio data;

[1797] A means for generating an avatar image using a GAN;

[1798] A means to adjust the quality of images generated by post-processing algorithms

[1799] A system including:

[1800] (Claim 2)

[1801] 2. The system of claim 1, wherein the speech features include formants and pitch.

[1802] (Claim 3)

[1803] 10. The system of claim 1, further comprising means for generating an image using a deep learning model.

[1804] "Application Example 1"

[1805] (Claim 1)

[1806] A voice input means;

[1807] means for transmitting audio data;

[1808] means for preprocessing the audio data;

[1809] means for extracting speech features;

[1810] means for selecting an avatar model based on speech features;

[1811] A means for generating an image based on an avatar model;

[1812] A means to check and adjust the quality of the generated avatar;

[1813] means for transmitting the generated avatar image data;

[1814] means for displaying an avatar image;

[1815] a means for receiving user feedback;

[1816] A means for generating assistant avatars to provide interactive shopping experiences in virtual stores

[1817] A system including:

[1818] (Claim 2)

[1819] 2. The system of claim 1, wherein the speech features include formants and pitch.

[1820] (Claim 3)

[1821] 10. The system of claim 1, further comprising means for generating an image using a deep learning model.

[1822] "Example 2: Combining Emotion Engines"

[1823] (Claim 1)

[1824] A voice input means;

[1825] means for transmitting audio data;

[1826] means for preprocessing the audio data;

[1827] means for extracting speech features;

[1828] means for recognizing emotions using an emotion engine;

[1829] means for selecting an avatar model based on speech features and emotions;

[1830] means for generating an image based on the avatar model and the emotion;

[1831] A means to check and adjust the quality of the generated avatar;

[1832] means for transmitting the generated avatar image data;

[1833] means for displaying an avatar image;

[1834] A way to receive user feedback

[1835] A system including:

[1836] (Claim 2)

[1837] 2. The system of claim 1, wherein the speech features include formants, pitch, tone, and rhythm.

[1838] (Claim 3)

[1839] 10. The system of claim 1, further comprising means for generating an image using a deep learning model.

[1840] "Application example 2 when combining emotion engines"

[1841] (Claim 1)

[1842] A voice input means;

[1843] means for transmitting audio data;

[1844] means for preprocessing the audio data;

[1845] means for extracting speech features;

[1846] A means of analyzing emotions,

[1847] means for selecting an avatar model based on speech features and emotions;

[1848] means for generating an image based on the avatar model and the emotion;

[1849] A means to check and adjust the quality of the generated avatar;

[1850] means for transmitting the generated avatar image data;

[1851] means for displaying an avatar image;

[1852] A way to receive user feedback

[1853] A system including:

[1854] (Claim 2)

[1855] 2. The system of claim 1, wherein the speech features include formants and pitch.

[1856] (Claim 3)

[1857] 10. The system of claim 1, further comprising means for generating an image using a deep learning model. [Explanation of symbols]

[1858] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A voice input means; means for transmitting audio data; means for preprocessing the audio data; means for extracting speech features; means for selecting an avatar model based on speech features; A means for generating an image based on an avatar model; A means to check and adjust the quality of the generated avatar; means for transmitting the generated avatar image data; means for displaying an avatar image; A way to receive user feedback A system including:

2. 2. The system of claim 1, wherein the speech features include formants and pitch.

3. The system of claim 1 , further comprising means for generating an image using a deep learning model.

Images