system

Abstract

We provide the system. [Solution] Means for receiving audio samples from the user, A means for analyzing received audio samples and generating an audio model, A method for registering the generated voice model as an NFT on the blockchain, A means for receiving real-time audio data and performing speech conversion by applying a generated audio model, A means of performing speech conversion processing using GPU resources in the cloud, A means of sending the voice-converted data back to the user in real time, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, with the progress of voice generation technology and blockchain technology, it has become possible to generate and manage the rights of individual voice models. However, in a system combining these technologies, there is a lack of a method for efficiently realizing a consistent process from the generation of a voice model to rights management and further to real-time voice conversion. In addition, when performing high-load processing required for real-time voice conversion, means for reducing the load and cost on the user's device are required.

Means for Solving the Problems

[0005] The present invention solves the above problems by the following means.

[0006] The system includes means for receiving audio samples from users, means for analyzing the received audio samples to generate an audio model, means for registering the generated audio model as an NFT on the blockchain, means for receiving real-time audio data and applying the generated audio model to perform audio conversion, means for executing audio conversion processing using GPU resources in the cloud, and means for returning the converted audio data to the user in real time.

[0007] This allows users to generate individual voice models and securely manage their rights, while also reducing the load and cost on devices by utilizing cloud-based resources for real-time voice conversion.

[0008] A "user" refers to an individual or organization that utilizes the system, provides voice samples, and receives the results of voice model generation and transformation.

[0009] A "voice sample" refers to a series of audio data that a user records and sends to the system, and serves as the foundational data for generating individual voice models.

[0010] A "voice model" is a digital representation generated using AI technology based on voice samples, and it is a dataset designed to mimic specific voice qualities and speaking styles.

[0011] "NFT" is an abbreviation for Non-Fungible Token, and it is a technology that uses blockchain technology to digitally prove ownership and usage rights of voice models.

[0012] Blockchain is a type of distributed ledger technology that uses multiple nodes to manage data in a distributed manner to prevent data tampering and fraud.

[0013] "Real-time audio data" refers to audio data that a user sends to the system in real time, and is intended to be converted immediately.

[0014] "Speech conversion" refers to the process of converting received audio data into audio data with a different format or characteristics, and is primarily performed by applying a speech model.

[0015] "GPU resources" refer to the resources of a Graphics Processing Unit, which are hardware resources used to perform advanced computational processing at high speed.

[0016] "Cloud" refers to remote data centers and computing resources provided via the internet, offering high-load processing and storage scalability.

[0017] "System" refers to a series of technical configurations in which multiple components or processes work together to provide a specific function. In this invention, it means the overall structure from receiving voice samples to generating and managing voice models and real-time voice conversion. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0020] First, the language used in the following description will be described.

[0021] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] The system of the present invention can create individual voice models using a voice generation AI, manage their rights using NFTs, and perform real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described below.

[0040] Server-side processing

[0041] User registration and receiving audio samples

[0042] The server first receives a user registration request. Once the user creates an account, the server stores this information in a database and generates a unique user ID. It then receives and temporarily stores a voice sample submitted by the user. The voice sample is used to understand the user's specific voice characteristics and speaking style.

[0043] Voice model generation and registration

[0044] The server analyzes the received audio sample and generates an audio model using AI technology. The generated audio model has user-specific characteristics and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[0045] Real-time speech conversion

[0046] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies a generated speech model to perform the conversion. During this process, GPU resources in the cloud are utilized to efficiently execute complex conversion operations. The converted audio data is then returned to the user in real time.

[0047] Terminal-side processing

[0048] Recording and sending audio samples

[0049] The user records an audio sample using the microphone on their smartphone or computer. Once the recording is complete, the device provides an interface for sending this data to the server.

[0050] Real-time speech conversion operation

[0051] When a user utilizes the real-time speech conversion function, the device streams audio data to the server in real time. After the server performs the conversion process, the device plays the converted audio in real time.

[0052] User-side operations

[0053] User registration and provision of audio samples

[0054] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[0055] Using real-time speech conversion

[0056] The user selects a conversion function and provides the audio in real time. The converted audio, processed on the server side, is played back on the device in real time and can be used directly by the user.

[0057] Specific example

[0058] When a user records an audio sample and sends it to the server

[0059] The user creates an account and records a 60-second audio sample following the application's instructions. Once recording is complete, the device sends the audio data to the server. The server analyzes the received audio sample and generates an audio model. The metadata of the audio model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[0060] When a user uses real-time speech conversion

[0061] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a generated speech model. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is sent back to the device in real time for the user to use.

[0062] This system provides comprehensive support for the entire process, from voice model generation to rights management and real-time voice conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[0063] The following describes the processing flow.

[0064] Server-side processing

[0065] Step 1:

[0066] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID.

[0067] Step 2:

[0068] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format.

[0069] Step 3:

[0070] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample, which are then input into the AI ​​model.

[0071] Step 4:

[0072] The server describes the generated voice model information as metadata and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[0073] Step 5:

[0074] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[0075] Step 6:

[0076] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion is performed by applying a speech model, generating audio data at high speed.

[0077] Step 7:

[0078] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[0079] Terminal-side processing

[0080] Step 1:

[0081] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[0082] Step 2:

[0083] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[0084] Step 3:

[0085] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[0086] Step 4:

[0087] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[0088] Step 5:

[0089] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[0090] User-side operations

[0091] Step 1:

[0092] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[0093] Step 2:

[0094] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[0095] Step 3:

[0096] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[0097] Step 4:

[0098] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[0099] (Example 1)

[0100] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0101] Existing voice conversion systems lack sufficient security and reliability in terms of the individuality and copyright management of voice data, and they have difficulty responding quickly to high-load voice conversion processing. Furthermore, there is a need for technology that can efficiently perform real-time voice conversion while protecting the copyright of voice data provided by users.

[0102] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0103] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model on a blockchain as a digital proof, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using remote computing resources, and means for returning the voice-converted data to the user in real time. This makes it possible to efficiently perform high-load voice conversion processing while ensuring the individuality of the voice data and copyright protection.

[0104] "Users" refer to individuals or companies that utilize the system, including those who provide voice samples or use real-time voice conversion.

[0105] A "voice sample" refers to data consisting of a user's own recorded voice, and is the basic data used to generate a voice model.

[0106] A "voice model" is a digital acoustic profile generated from analyzed voice samples, and refers to a model that possesses specific voice characteristics or speaking style traits.

[0107] "Digital proof means" refers to a technology that registers the metadata of a generated voice model on a distributed ledger technology (blockchain) to digitally prove ownership and usage rights.

[0108] "Blockchain" refers to a distributed ledger technology composed of multiple nodes, designed to ensure data transparency and prevent tampering.

[0109] "Real-time audio data" refers to audio data provided by users in real time, which is immediately converted into speech.

[0110] "Remote computing resources" refer to high-performance computing resources (e.g., GPUs) located in the cloud, which are used to efficiently distribute and execute high-load computational processes such as speech conversion.

[0111] "Speech conversion processing" refers to the process of applying a generated speech model to received real-time speech data and converting it into different speech characteristics.

[0112] The system of the present invention utilizes a speech generation AI to create individual speech models, manages their rights as digital proof-of-services (NFTs), and performs real-time speech conversion using cloud-based remote computing resources. This embodiment includes the processes of receiving speech samples, generating and registering speech models, and real-time speech conversion.

[0113] Server-side processing

[0114] The server first receives a voice sample from the user. The received voice sample is temporarily stored and analyzed in detail using AI technology. Deep learning frameworks such as TENSORFLOW® and PyTorch are used for this analysis. As a result of the analysis, a user-specific voice model is generated. This voice model is described as metadata and registered on the blockchain as a means of digital proof. Distributed ledger technology is used as the blockchain platform. This digitally proves ownership and usage rights of the voice model.

[0115] Next, in the real-time speech conversion process, the user sends voice data to the server in real time via their device. The server analyzes the received voice data and performs speech conversion by applying a pre-generated voice model. During this process, a high-performance GPU in the cloud (such as NVIDIA Tesla V100) is used to efficiently perform the speech conversion. The converted voice data is then returned to the user in real time.

[0116] Terminal-side processing

[0117] The device provides an interface for recording user-provided audio samples and sending them to the server. When the user records an audio sample and presses the send button, this audio data is sent to the server. Furthermore, when the user utilizes the real-time voice conversion function, the device streams the audio data in real time via the microphone and plays the converted audio in real time.

[0118] User-side operations

[0119] Users create an account through the application and proceed to a page for providing a voice sample. By recording and submitting a voice sample according to the instructions, they register their voice with the system. They can also select the real-time voice conversion function and provide their voice in real time. The converted voice, processed on the server side, is played back on the device in real time and can be used directly by the user.

[0120] Specific example

[0121] 1. Prompt for user registration and voice sample provision

[0122] Launch the application and create an account. Then, follow the instructions to record a 60-second voice sample.

[0123] 2. Prompts for real-time speech conversion

[0124] To use the real-time speech conversion feature, prepare your microphone and press the conversion button in the application. Your speech will be converted in real time.

[0125] As described above, the system of the present invention provides a series of processes in which a user provides a voice sample, a voice model is generated and registered based on it, and voice conversion is performed in real time. This makes it possible to achieve efficient voice conversion while ensuring the security and reliability of voice data.

[0126] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0127] Step 1:

[0128] The user enters information such as their name, email address, and password on the account creation screen and presses the submit button. The entered information is sent from the device to the server. The server stores the received information in a database and generates a unique user ID. This is to manage the user's basic information, and the database includes the user ID, name, email address, etc. The user ID generated by the server is output.

[0129] Step 2:

[0130] The user proceeds to the audio sample provision page and records an audio sample using their device's microphone. Once recording is complete, the user presses the send button, and the audio data is sent from the device to the server. The server receives the audio data and temporarily stores it. This audio data is the basis for analyzing the user's voice quality and speaking style. An output confirms that the received audio data has been saved to the server.

[0131] Step 3:

[0132] The server analyzes the stored audio samples and generates an audio model using AI technology. Specifically, it extracts audio features using deep learning frameworks such as TensorFlow and PyTorch to generate an acoustic model. This model has user-specific audio characteristics. The generated audio model is output as a result of the analysis.

[0133] Step 4:

[0134] The server creates metadata for the generated voice model and registers it on the blockchain as a digital proof of value (NFT). A smart contract is used in this process, and the voice model's metadata is recorded on the blockchain platform (e.g., Ethereum or Hyperledger). This digitally proves ownership and usage rights of the voice model. The generated voice model's metadata and registration completion information are output.

[0135] Step 5:

[0136] If a user desires real-time speech conversion, they launch the application on their device and provide real-time audio data through the microphone. The device streams the real-time audio data to the server. The server receives this audio data and applies a pre-generated speech model to perform real-time speech conversion. This process is efficiently performed using a high-performance GPU in the cloud (e.g., NVIDIA Tesla V100). The converted audio data is output and immediately returned to the user.

[0137] Step 6:

[0138] Once the converted audio data is sent back to the device, the device plays this data in real time. The user can immediately listen to the converted audio. This process continues as long as the user continues to provide audio in real time. The output converted audio data is played back to the user in real time.

[0139] (Application Example 1)

[0140] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0141] This invention relates to a system that uses a speech generation AI to create individual speech models and converts speech in real time based on those models. Conventional technologies have problems with inefficient processing, requiring significant computational resources and time for creating speech models and real-time speech conversion. Furthermore, the rights management of speech models is ambiguous, creating risks of theft and misuse. In addition, it has been difficult for users to control virtual characters with their own voices. This invention aims to solve these problems and provide a highly efficient and secure speech conversion system.

[0142] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0143] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This ensures reliable rights management of the voice model and enables high-speed and highly efficient real-time voice conversion. Furthermore, the generated voice model enables a virtual character to perform in real time using the user's voice.

[0144] "Means for receiving audio samples from users" refers to means for recording audio data provided by users and sending it to the server.

[0145] "Means for analyzing received audio samples and generating audio models" refers to a method by which a server analyzes received audio data and constructs individual audio models based on that data.

[0146] "Method for registering a generated voice model as an NFT on the blockchain" refers to a method for recording the metadata of a generated voice model on the blockchain as an NFT and digitally proving the rights to it.

[0147] "Means for receiving real-time audio data and performing audio conversion by applying a generated audio model" refers to means for receiving audio data provided by the user in real time and performing conversion processing by applying an existing audio model.

[0148] "A means of performing speech conversion processing using GPU resources on the cloud" refers to a method of efficiently distributing the computational load of speech conversion processing by utilizing GPU resources located in a cloud environment.

[0149] "A means of sending the converted audio data back to the user in real time" refers to a means of immediately delivering the converted audio data to the user's device.

[0150] "A means of enabling a virtual character to perform in real time using a user's voice by utilizing a generated voice model" refers to a method of enabling a virtual character to play back the user's voice in real time using live streaming or recording, by leveraging a generated voice model.

[0151] This invention relates to a system that generates individual voice models using voice samples provided by the user and performs real-time voice conversion using these models. The overall system configuration and specific embodiments are described below.

[0152] Server-side processing

[0153] User registration and receiving audio samples

[0154] The server accepts users creating an account and recording and submitting a voice sample. The user's account information and voice sample are stored in the server's database, and a unique user ID is generated. This process identifies each individual user.

[0155] Voice model generation and NFT registration

[0156] The received audio samples are analyzed using AI technology to generate individual voice models. These voice models reproduce the user's specific voice characteristics and speaking style. Furthermore, the metadata of these generated voice models is registered on the blockchain as NFTs, digitally proving the copyright and usage rights of the voice models. Smart contract technology is used to ensure the secure management of these rights.

[0157] Real-time speech conversion

[0158] When a user utilizes the real-time speech conversion function, the server receives streaming audio data from the user and applies a generated speech model to perform the conversion. This process leverages GPU resources in the cloud to achieve high-speed and efficient speech conversion. The converted audio data is returned to the user in real time.

[0159] Terminal-side processing

[0160] Recording and sending audio samples

[0161] The terminal provides an interface for the user to record voice samples. The recorded voice samples are sent to a server and used for analysis and voice model generation.

[0162] Real-time speech conversion operation

[0163] When a user utilizes the real-time voice conversion function, the device acquires voice data via the microphone and streams it to the server. After the voice conversion process on the server is complete, the converted audio is played back in real time from the device.

[0164] User-side operations

[0165] User registration and provision of audio samples

[0166] Users create an account using the application, record a voice sample, and send it to the system. At this stage, a voice model is generated and registered on the blockchain as an NFT.

[0167] Using real-time speech conversion

[0168] When a user utilizes the real-time voice conversion function, the device sends audio to the server in real time and receives the converted audio in real time. This allows users to apply their own voice to virtual characters and perform with them.

[0169] Program Processing Description

[0170] This system uses Python, the requests library, and the sounddevice library to record, send, and perform real-time speech conversion of audio samples. On the server side, cloud-based GPU resources are used to process the speech conversion at high speed. The generated speech models are then registered on the blockchain as NFTs using smart contracts.

[0171] Specific example

[0172] For example, when a user uses a virtual performance application, the following process takes place:

[0173] 1. The user creates an account, records an audio sample according to the instructions, and sends it to the server.

[0174] 2. The server receives the voice sample, analyzes it, and generates a voice model.

[0175] 3. The voice model will be registered on the blockchain as an NFT.

[0176] 4. Users utilize a real-time voice conversion function to apply their own voice to a virtual character and perform.

[0177] Example of a prompt:

[0178] This program records a voice sample provided by the user, sends it to an API server to generate a voice model, and further includes the process of converting and playing back the voice in real time. Below is an example of the code that records the user's voice and sends the sample to the server.

[0179] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0180] Step 1:

[0181] User registration and recording and submission of voice samples.

[0182] The user accesses the application using their device and creates an account. Once the user enters and submits their account information (username, password, etc.), the server receives it and stores it in its database. Next, the user records a voice sample using their device's microphone and sends the data to the server. The server receives this voice sample and temporarily stores it for analysis.

[0183] Input: User account information and voice sample

[0184] Output: User ID and voice sample data

[0185] Step 2:

[0186] Voice model generation and NFT registration

[0187] The server analyzes the received audio sample and generates a voice model using an AI model based on that data. This voice model reproduces the user's specific voice quality and speaking style. The metadata of the generated voice model is registered on the blockchain as an NFT, where digital copyright management takes place.

[0188] Input: Audio sample data

[0189] Output: Voice model registered as an NFT

[0190] Step 3:

[0191] Preparing for real-time speech conversion

[0192] The user selects the real-time speech conversion function using their device. The device uses its microphone to acquire voice data in real time and prepares to send it to the server. At this time, the server prepares to use the already generated voice model.

[0193] Input: User's real-time voice data, existing voice model

[0194] Output: Real-time streaming start notification

[0195] Step 4:

[0196] Receiving and converting real-time audio data

[0197] The terminal sends real-time audio data from the user to the server in streaming format. The server receives this audio data and applies a generative AI model to perform real-time speech conversion. This process utilizes GPU resources in the cloud to achieve high-speed and efficient speech conversion.

[0198] Input: Real-time audio data sent by the user

[0199] Output: Converted audio data

[0200] Step 5:

[0201] Return and playback of the converted audio data

[0202] The server immediately returns the converted audio data to the user's terminal. The terminal plays this received data in real time, allowing the user to instantly check the converted audio. This process enables the virtual character to perform in real time using the user's voice.

[0203] Input: Converted audio data

[0204] Output: Audio played back in real time

[0205] Through these steps, individual voice models are generated based on voice samples provided by the user, and these models are used to achieve real-time voice conversion. Furthermore, the copyright of the generated voice models can be registered as NFTs on the blockchain, allowing for digital rights management.

[0206] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0207] The system of this invention creates individual voice models using a voice generation AI and an emotion engine, manages their rights using NFTs, and performs real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described in detail below.

[0208] Server-side processing

[0209] User registration and receiving audio samples

[0210] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[0211] Voice model generation, registration, and sentiment analysis

[0212] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific characteristics and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[0213] Real-time speech conversion and emotion recognition

[0214] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results from the sentiment engine to perform the conversion. In this process, GPU resources in the cloud are utilized to efficiently execute the complex conversion process. The converted audio data is returned to the user in real time.

[0215] Terminal-side processing

[0216] Recording and sending audio samples

[0217] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[0218] Real-time speech conversion operation

[0219] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[0220] User-side operations

[0221] User registration and provision of audio samples

[0222] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[0223] Using real-time speech conversion

[0224] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[0225] Specific example

[0226] When a user records an audio sample and sends it to the server

[0227] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates a voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[0228] When a user uses real-time speech conversion

[0229] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[0230] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[0231] The following describes the processing flow.

[0232] Server-side processing

[0233] Step 1:

[0234] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID.

[0235] Step 2:

[0236] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format. After saving, a confirmation message is sent to the user.

[0237] Step 3:

[0238] The server analyzes the received audio sample and generates an audio model using an AI model. This analysis process extracts feature elements from the audio sample.

[0239] Step 4:

[0240] The server uses an emotion engine to analyze the user's emotions from the voice sample. The results of the emotion analysis are reflected in the voice model.

[0241] Step 5:

[0242] The server describes the metadata of the generated voice model and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[0243] Step 6:

[0244] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[0245] Step 7:

[0246] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion process is performed by applying the speech model and sentiment analysis results, generating audio data at high speed.

[0247] Step 8:

[0248] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[0249] Terminal-side processing

[0250] Step 1:

[0251] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[0252] Step 2:

[0253] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[0254] Step 3:

[0255] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[0256] Step 4:

[0257] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[0258] Step 5:

[0259] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[0260] User-side operations

[0261] Step 1:

[0262] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[0263] Step 2:

[0264] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[0265] Step 3:

[0266] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[0267] Step 4:

[0268] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[0269] (Example 2)

[0270] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0271] In recent years, speech generation technology has advanced dramatically, but there is a need for a method to efficiently generate speech models that reflect the individual voice quality and emotions of each user, and to clearly manage their ownership and usage rights. Furthermore, real-time, high-quality speech conversion requires significant computing resources, necessitating systems that can process this efficiently.

[0272] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to extract features and generate a voice model, means for analyzing the user's emotions from the voice data and reflecting the results in the voice model, means for registering the metadata of the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model and emotion analysis results, means for executing high-load voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This enables the efficient generation of voice models that reflect the user's individual voice quality and emotions, and high-quality voice conversion in real time.

[0273] A "user" is a person or organization that uses the system and is the entity that performs operations such as providing voice samples or transmitting voice data in real time.

[0274] A "voice sample" is audio data that a user records and provides to the system, and is used to understand the user's voice quality and speaking style.

[0275] "Features" are acoustic characteristics extracted from audio data and are used to generate audio models. Examples include Mel-frequency cepstrum coefficients (MFCCs).

[0276] A "voice model" is a model generated by analyzing received voice samples and based on their features, representing a digital voice that reflects the user's voice quality and emotions.

[0277] "Emotion analysis" is the process of estimating a user's emotional state from voice data, and is performed using an emotion engine.

[0278] "NFT" is an abbreviation for Non-Fungible Token, which is a technology used to prove the ownership and usage rights of digital data on the blockchain.

[0279] "Blockchain" is a mechanism for securely and distributively managing digital data. It is a technology that records data in a series of "blocks" and links them together in a chain.

[0280] "Real-time voice data" refers to voice data provided in real-time by a user through a microphone, which is the target to be sent to a server for real-time analysis and conversion.

[0281] "GPU resources" refer to the computing power of a graphics processing unit, which are resources used to efficiently execute high-load computing processes.

[0282] "Voice conversion" is a process of analyzing received real-time voice data and applying the generated voice model and sentiment analysis results for conversion.

[0283] "Metadata" is supplementary material containing information about a voice model, which is data registered on the blockchain as an NFT.

[0284] The system of the present invention combines voice generation technology and sentiment analysis technology, registers it as an NFT, and performs voice conversion in real-time. The overall configuration and specific embodiments of the system will be described in detail below.

[0285] Server-side processing

[0286] The server performs the following main processes.

[0287] 1. User registration and reception of voice samples

[0288] The server receives account creation requests from users and stores the entered information in a database (e.g., MySQL® or PostgreSQL). It generates a user ID, receives a voice sample, and temporarily stores it in cloud storage (e.g., Amazon S3 or Google® Cloud Storage).

[0289] 2. Generation of voice models and sentiment analysis

[0290] The server uses a speech analysis library (e.g., Librosa or PyDub) to analyze the received audio sample and extract features. Based on the extracted features, it uses a generative AI model (e.g., TensorFlow or PyTorch) to generate an audio model. An emotion engine (e.g., EmotionAI) analyzes the emotions from the audio data and incorporates the results into the audio model.

[0291] 3. Register the voice model as an NFT.

[0292] The server describes the metadata of the generated voice model and registers it on a blockchain (e.g., Ethereum or Solana) using a smart contract. This digitally proves ownership and usage rights of the voice model.

[0293] 4. Real-time speech conversion and emotion recognition

[0294] The server receives real-time audio data from the user, analyzes the data, applies a generated audio model and sentiment analysis results, and performs speech conversion. This process utilizes GPU resources in the cloud (e.g., NVIDIA GPU Cloud). The converted audio data is returned to the user in real time.

[0295] Terminal-side processing

[0296] The terminal performs the following main processes:

[0297] 1. Recording and Transmission of Audio Samples

[0298] The terminal provides the user with an account creation screen. When the user enters the necessary information and presses the send button, the information is sent to the server. The user records an audio sample, and the data is temporarily saved. Once the recording is complete, it is sent to the server.

[0299] 2. Operation of Real - Time Audio Conversion

[0300] The terminal has a real - time audio conversion function for the user. When the user provides audio in real - time, the audio data is streamed to the server. After the conversion process is performed on the server side, the converted audio data is played back on the terminal.

[0301] User - side Operations

[0302] The user performs the following main operations.

[0303] 1. User Registration and Provision of Audio Samples

[0304] The user creates an account through the application and proceeds to the page for providing audio samples. By recording an audio sample and providing it to the system, the user registers their voice.

[0305] 2. Use of Real - Time Audio Conversion

[0306] The user selects the real - time audio conversion function within the application and provides audio in real - time towards the microphone. The audio data processed on the server side recognizes the emotional state and is played back in real - time from the terminal as appropriately adjusted audio.

[0307] Specific Examples

[0308] Examples of specific processing procedures are shown below.

[0309] When a user records an audio sample and sends it to the server

[0310] 1. The user enters their name, email address, password, etc. on the account creation screen and presses the submit button.

[0311] 2. The server saves the received information to the database and generates a unique user ID.

[0312] 3. The user follows the application's instructions and records a 60-second audio sample via the microphone.

[0313] 4. After recording is complete, the device uploads the audio file to the server.

[0314] 5. The server analyzes the audio data and generates an audio model.

[0315] 6. The emotion engine analyzes the audio data and incorporates the emotional information into the audio model.

[0316] 7. The server describes the metadata of the voice model and registers it on the blockchain.

[0317] 8. A completion notification is sent to the user.

[0318] When a user uses real-time speech conversion

[0319] 1. The user launches the application and selects the real-time speech conversion function.

[0320] 2. The device captures the user's real-time voice through the microphone and streams it to the server.

[0321] 3. The server analyzes the received audio data and performs speech conversion by applying the audio model and sentiment analysis results.

[0322] 4. The converted audio data is efficiently processed using GPU resources in the cloud.

[0323] 5. The converted audio data is sent back to the device in real time and played back through the speaker or headphones.

[0324] Examples of input prompt messages

[0325] "Please register a new account. Enter the required information and record and submit a 60-second voice sample."

[0326] "Please begin real-time speech conversion. When you speak into the microphone, your voice will be automatically sent to the server."

[0327] This system utilizes generative AI models and an emotion engine to consistently support the entire process, from generating voice models to rights management, emotion analysis, and real-time speech conversion. Furthermore, it leverages cloud-based GPU resources to efficiently perform advanced processing.

[0328] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0329] Step 1:

[0330] The user enters the required information on the account creation screen and presses the submit button. This information includes name, email address, and password. The server receives this information and stores it in the database. Based on the information stored in the database, the server generates a unique user ID and stores this user ID in the database. The input data is the user's registration information, and the output is the user ID stored in the database.

[0331] Step 2:

[0332] The user proceeds to a screen to record a voice sample. The user presses the record button to record a voice sample through the microphone. Once recording is complete, the device temporarily saves the voice sample data. The input is the user's voice data, and the output is the temporarily saved voice data.

[0333] Step 3:

[0334] The device sends temporarily stored audio data to the server. At the same time, metadata such as the audio file format (e.g., WAV format) and sampling rate is also sent. The server saves the received audio sample to cloud storage. The data received by the server is the audio sample, and the output is the audio data stored in cloud storage.

[0335] Step 4:

[0336] The server analyzes the audio sample and extracts features. Specifically, it uses an audio analysis library (e.g., Librosa or PyDub) to extract acoustic features. The extracted features include MFCC (Mel-frequency cepstrum coefficients) and zero crossover ratio. The input data is an audio sample, and the output is the extracted features.

[0337] Step 5:

[0338] The server generates a speech model using a generative AI model based on the extracted features. The speech model is constructed using AI model libraries such as TensorFlow or PyTorch. The input data consists of features, and the output is the speech model.

[0339] Step 6:

[0340] The server analyzes the user's emotions from the audio data. An emotion engine (e.g., EmotionAI) is used to estimate the emotional state within the audio data. The analysis results are output as emotion labels such as joy, sadness, and anger. The input is audio data, and the output is emotion labels.

[0341] Step 7:

[0342] The server incorporates the emotion analysis results into the speech model. By incorporating emotion labels into the speech model, it becomes possible to generate speech that reflects the user's emotional state. The input data consists of the speech model and emotion labels, and the output is the speech model that reflects the emotions.

[0343] Step 8:

[0344] The server describes the metadata of the generated voice model and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model. The input is the metadata of the voice model, and the output is an NFT registered on the blockchain.

[0345] Step 9:

[0346] When a user uses the real-time speech conversion function, the device captures audio data in real time and streams it to the server. The input is real-time audio data, and the output is streamed audio data.

[0347] Step 10:

[0348] The server analyzes real-time audio data it receives and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, high-load processing is efficiently executed using GPU resources in the cloud. The input data consists of real-time audio data and a speech model, while the output is the converted audio data.

[0349] Step 11:

[0350] The server sends the converted audio data back to the user in real time. The terminal plays the received converted audio data through its speaker or headphones. The input is the converted audio data, and the output is the played audio.

[0351] This ensures that specific data processing and calculations are performed at each processing step, ultimately resulting in a system that allows users to smoothly utilize voice conversion.

[0352] (Application Example 2)

[0353] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0354] In traditional online shopping, it was difficult to provide personalized voice models or emotionally resonant voices for virtual assistants when they interacted with users. This resulted in a mechanical user experience, leading to decreased emotional satisfaction and user engagement. Furthermore, there was a lack of efficient methods for handling the high-load processing required for real-time voice conversion.

[0355] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0356] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate individual voice models, means for registering the generated voice models as NFTs on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice models and sentiment analysis, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time and appropriately adjusting the voice based on the user's emotional state. This enables interactive and emotional responses by a personalized voice assistant in virtual stores, improving the user experience. Furthermore, it allows for the efficient distribution and execution of high-load cloud-based processing.

[0357] "Means for receiving voice samples from users" refers to devices or software that provide an interface for users to record their own voice samples and send them to the system.

[0358] "Means for analyzing received audio samples and generating individual audio models" refers to devices or algorithms that extract feature quantities from audio samples and generate user-specific audio models.

[0359] "Methods for registering generated voice models as NFTs on the blockchain" refers to devices or software that generate metadata for a voice model as an NFT and register it on the blockchain using smart contracts.

[0360] "Means for receiving real-time voice data and performing voice conversion by applying a generated voice model and sentiment analysis" refers to a device or software that receives real-time voice data from a user and converts it using a user-specific voice model and sentiment analysis engine.

[0361] "Means of performing speech conversion processing using cloud-based GPU resources" refers to devices or software that efficiently perform the high-load processing of real-time speech conversion by utilizing cloud-based high-performance GPU resources.

[0362] "Means for returning converted audio data to the user in real time and appropriately adjusting the audio based on the user's emotional state" refers to a device or software that immediately returns converted audio data to the user and further appropriately adjusts the audio based on the user's emotional state.

[0363] This invention realizes a system in which a virtual assistant with an individualized voice model guides users while they enjoy shopping in a virtual store. The overall system configuration and specific embodiments are described in detail below.

[0364] Server-side processing

[0365] User registration and receiving audio samples

[0366] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[0367] Voice model generation, registration, and sentiment analysis

[0368] The server analyzes the received audio sample and generates an individual audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion analysis engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific features and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[0369] Real-time speech conversion and emotion recognition

[0370] Real-time speech-to-text conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results to perform the conversion. During this process, GPU resources in the cloud (e.g., Google Cloud Speech-to-Text) are utilized to efficiently execute complex conversion processes. The converted audio data is then returned to the user in real time.

[0371] Terminal-side processing

[0372] Recording and sending audio samples

[0373] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[0374] Real-time speech conversion operation

[0375] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[0376] User-side operations

[0377] User registration and provision of audio samples

[0378] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[0379] Using real-time speech conversion

[0380] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[0381] Specific example

[0382] When a user records an audio sample and sends it to the server

[0383] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates an individual voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[0384] When a user uses real-time speech conversion

[0385] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[0386] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[0387] Examples of prompt statements

[0388] For example, if a user asks the virtual assistant, "Tell me more about this product," the following prompt will be generated.

[0389] The user asked, "Tell me more about this product."

[0390] Emotion analysis result: Very interesting

[0391] Generated audio: An audio that enthusiastically explains detailed information.

[0392] This prompt allows for the generation of appropriate messages in real time, based on sentiment analysis results, and provides them to the user.

[0393] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0394] Step 1:

[0395] To register an account, users access the account creation screen through the application, enter the required information, and submit it.

[0396] Input: User information (name, email address, password, etc.)

[0397] Output: Unique User ID

[0398] Specific operation: The terminal sends the entered user information to the server, which stores this information in a database and generates a unique user ID.

[0399] Step 2:

[0400] The user records an audio sample. Once recording is complete, the device sends the audio sample data to the server.

[0401] Input: User voice sample

[0402] Output: Temporarily saved audio sample

[0403] Specific operation: The terminal records an audio sample, temporarily stores it, and then provides an interface for sending it to the server. The server receives the audio sample and temporarily stores it.

[0404] Step 3:

[0405] The server analyzes the received audio samples and generates individual audio models. During this process, it uses AI models to extract feature vectors from the audio samples.

[0406] Input: Audio sample

[0407] Output: Individual speech models

[0408] Specific operation: The server uses an AI library for speech generation to analyze speech samples, extract features, and then generate individual speech models.

[0409] Step 4:

[0410] The metadata of the generated voice model is registered on the blockchain as an NFT.

[0411] Input: Voice model

[0412] Output: NFT-registered voice models

[0413] Specific operation: The server generates voice model information as an NFT and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model.

[0414] Step 5:

[0415] To utilize the real-time voice conversion function, the user provides voice data via the microphone. The device then streams this real-time voice data to the server.

[0416] Input: Real-time audio data

[0417] Output: Streamed audio data

[0418] Specific operation: The device captures audio data in real time and streams it to the server.

[0419] Step 6:

[0420] The server analyzes the received real-time audio data and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, it utilizes GPU resources in the cloud to perform high-speed conversion.

[0421] Input: Real-time audio data, audio model, sentiment analysis results

[0422] Output: Converted audio data

[0423] Specific operation: The server uses cloud-based GPU resources to analyze and convert real-time audio data, and then applies the sentiment analysis results to adjust the audio.

[0424] Step 7:

[0425] The converted audio data is sent back to the user in real time, and the audio is appropriately adjusted based on the user's emotional state.

[0426] Input: Converted audio data, user's emotional state

[0427] Output: Real-time adjusted converted audio

[0428] Specific operation: The server immediately returns the converted audio data to the user and adjusts the audio appropriately according to the user's emotional state. The terminal receives this data and plays it back in real time through speakers or headphones.

[0429] In this way, by sequentially executing the specific processes performed at each step, the virtual assistant can provide a personalized voice model and an interactive user experience based on sentiment analysis.

[0430] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0431] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0432] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0433] [Second Embodiment]

[0434] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0435] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0436] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0437] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0438] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0440] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0441] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0442] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0443] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0444] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0445] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0446] The system of the present invention can create individual voice models using a voice generation AI, manage their rights using NFTs, and perform real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described below.

[0447] Server-side processing

[0448] User registration and receiving audio samples

[0449] The server first receives a user registration request. Once the user creates an account, the server stores this information in a database and generates a unique user ID. It then receives and temporarily stores a voice sample submitted by the user. The voice sample is used to understand the user's specific voice characteristics and speaking style.

[0450] Voice model generation and registration

[0451] The server analyzes the received audio sample and generates an audio model using AI technology. The generated audio model has user-specific characteristics and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[0452] Real-time speech conversion

[0453] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies a generated speech model to perform the conversion. During this process, GPU resources in the cloud are utilized to efficiently execute complex conversion operations. The converted audio data is then returned to the user in real time.

[0454] Terminal-side processing

[0455] Recording and sending audio samples

[0456] The user records an audio sample using the microphone on their smartphone or computer. Once the recording is complete, the device provides an interface for sending this data to the server.

[0457] Real-time speech conversion operation

[0458] When a user utilizes the real-time speech conversion function, the device streams audio data to the server in real time. After the server performs the conversion process, the device plays the converted audio in real time.

[0459] User-side operations

[0460] User registration and provision of audio samples

[0461] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[0462] Using real-time speech conversion

[0463] The user selects a conversion function and provides the audio in real time. The converted audio, processed on the server side, is played back on the device in real time and can be used directly by the user.

[0464] Specific example

[0465] When a user records an audio sample and sends it to the server

[0466] The user creates an account and records a 60-second audio sample following the application's instructions. Once recording is complete, the device sends the audio data to the server. The server analyzes the received audio sample and generates an audio model. The metadata of the audio model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[0467] When a user uses real-time speech conversion

[0468] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a generated speech model. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is sent back to the device in real time for the user to use.

[0469] This system provides comprehensive support for the entire process, from voice model generation to rights management and real-time voice conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[0470] The following describes the processing flow.

[0471] Server-side processing

[0472] Step 1:

[0473] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID.

[0474] Step 2:

[0475] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format.

[0476] Step 3:

[0477] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample, which are then input into the AI ​​model.

[0478] Step 4:

[0479] The server describes the generated voice model information as metadata and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[0480] Step 5:

[0481] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[0482] Step 6:

[0483] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion is performed by applying a speech model, generating audio data at high speed.

[0484] Step 7:

[0485] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[0486] Terminal-side processing

[0487] Step 1:

[0488] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[0489] Step 2:

[0490] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[0491] Step 3:

[0492] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[0493] Step 4:

[0494] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[0495] Step 5:

[0496] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[0497] User-side operations

[0498] Step 1:

[0499] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[0500] Step 2:

[0501] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[0502] Step 3:

[0503] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[0504] Step 4:

[0505] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[0506] (Example 1)

[0507] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0508] Existing voice conversion systems lack sufficient security and reliability in terms of the individuality and copyright management of voice data, and they have difficulty responding quickly to high-load voice conversion processing. Furthermore, there is a need for technology that can efficiently perform real-time voice conversion while protecting the copyright of voice data provided by users.

[0509] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0510] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model on a blockchain as a digital proof, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using remote computing resources, and means for returning the voice-converted data to the user in real time. This makes it possible to efficiently perform high-load voice conversion processing while ensuring the individuality of the voice data and copyright protection.

[0511] "Users" refer to individuals or companies that utilize the system, including those who provide voice samples or use real-time voice conversion.

[0512] A "voice sample" refers to data consisting of a user's own recorded voice, and is the basic data used to generate a voice model.

[0513] A "voice model" is a digital acoustic profile generated from analyzed voice samples, and refers to a model that possesses specific voice characteristics or speaking style traits.

[0514] "Digital proof means" refers to a technology that registers the metadata of a generated voice model on a distributed ledger technology (blockchain) to digitally prove ownership and usage rights.

[0515] "Blockchain" refers to a distributed ledger technology composed of multiple nodes, designed to ensure data transparency and prevent tampering.

[0516] "Real-time audio data" refers to audio data provided by users in real time, which is immediately converted into speech.

[0517] "Remote computing resources" refer to high-performance computing resources (e.g., GPUs) located in the cloud, which are used to efficiently distribute and execute high-load computational processes such as speech conversion.

[0518] "Speech conversion processing" refers to the process of applying a generated speech model to received real-time speech data and converting it into different speech characteristics.

[0519] The system of the present invention utilizes a speech generation AI to create individual speech models, manages their rights as digital proof-of-services (NFTs), and performs real-time speech conversion using cloud-based remote computing resources. This embodiment includes the processes of receiving speech samples, generating and registering speech models, and real-time speech conversion.

[0520] Server-side processing

[0521] The server first receives a voice sample from the user. The received voice sample is temporarily stored and analyzed in detail using AI technology. Deep learning frameworks such as TensorFlow and PyTorch are used for this analysis. As a result of the analysis, a user-specific voice model is generated. This voice model is described as metadata and registered on the blockchain as a means of digital proof. Distributed ledger technology is used as the blockchain platform. This digitally proves ownership and usage rights of the voice model.

[0522] Next, in the real-time speech conversion process, the user sends voice data to the server in real time via their device. The server analyzes the received voice data and performs speech conversion by applying a pre-generated voice model. During this process, a high-performance GPU in the cloud (such as NVIDIA Tesla V100) is used to efficiently perform the speech conversion. The converted voice data is then returned to the user in real time.

[0523] Terminal-side processing

[0524] The device provides an interface for recording user-provided audio samples and sending them to the server. When the user records an audio sample and presses the send button, this audio data is sent to the server. Furthermore, when the user utilizes the real-time voice conversion function, the device streams the audio data in real time via the microphone and plays the converted audio in real time.

[0525] User-side operations

[0526] Users create an account through the application and proceed to a page for providing a voice sample. By recording and submitting a voice sample according to the instructions, they register their voice with the system. They can also select the real-time voice conversion function and provide their voice in real time. The converted voice, processed on the server side, is played back on the device in real time and can be used directly by the user.

[0527] Specific example

[0528] 1. Prompt for user registration and voice sample provision

[0529] Launch the application and create an account. Then, follow the instructions to record a 60-second voice sample.

[0530] 2. Prompts for real-time speech conversion

[0531] To use the real-time speech conversion feature, prepare your microphone and press the conversion button in the application. Your speech will be converted in real time.

[0532] As described above, the system of the present invention provides a series of processes in which a user provides a voice sample, a voice model is generated and registered based on it, and voice conversion is performed in real time. This makes it possible to achieve efficient voice conversion while ensuring the security and reliability of voice data.

[0533] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0534] Step 1:

[0535] The user enters information such as their name, email address, and password on the account creation screen and presses the submit button. The entered information is sent from the device to the server. The server stores the received information in a database and generates a unique user ID. This is to manage the user's basic information, and the database includes the user ID, name, email address, etc. The user ID generated by the server is output.

[0536] Step 2:

[0537] The user proceeds to the audio sample provision page and records an audio sample using their device's microphone. Once recording is complete, the user presses the send button, and the audio data is sent from the device to the server. The server receives the audio data and temporarily stores it. This audio data is the basis for analyzing the user's voice quality and speaking style. An output confirms that the received audio data has been saved to the server.

[0538] Step 3:

[0539] The server analyzes the stored audio samples and generates an audio model using AI technology. Specifically, it extracts audio features using deep learning frameworks such as TensorFlow and PyTorch to generate an acoustic model. This model has user-specific audio characteristics. The generated audio model is output as a result of the analysis.

[0540] Step 4:

[0541] The server creates metadata for the generated voice model and registers it on the blockchain as a digital proof of value (NFT). A smart contract is used in this process, and the voice model's metadata is recorded on the blockchain platform (e.g., Ethereum or Hyperledger). This digitally proves ownership and usage rights of the voice model. The generated voice model's metadata and registration completion information are output.

[0542] Step 5:

[0543] If a user desires real-time speech conversion, they launch the application on their device and provide real-time audio data through the microphone. The device streams the real-time audio data to the server. The server receives this audio data and applies a pre-generated speech model to perform real-time speech conversion. This process is efficiently performed using a high-performance GPU in the cloud (e.g., NVIDIA Tesla V100). The converted audio data is output and immediately returned to the user.

[0544] Step 6:

[0545] Once the converted audio data is sent back to the device, the device plays this data in real time. The user can immediately listen to the converted audio. This process continues as long as the user continues to provide audio in real time. The output converted audio data is played back to the user in real time.

[0546] (Application Example 1)

[0547] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0548] This invention relates to a system that uses a speech generation AI to create individual speech models and converts speech in real time based on those models. Conventional technologies have problems with inefficient processing, requiring significant computational resources and time for creating speech models and real-time speech conversion. Furthermore, the rights management of speech models is ambiguous, creating risks of theft and misuse. In addition, it has been difficult for users to control virtual characters with their own voices. This invention aims to solve these problems and provide a highly efficient and secure speech conversion system.

[0549] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0550] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This ensures reliable rights management of the voice model and enables high-speed and highly efficient real-time voice conversion. Furthermore, the generated voice model enables a virtual character to perform in real time using the user's voice.

[0551] "Means for receiving audio samples from users" refers to means for recording audio data provided by users and sending it to the server.

[0552] "Means for analyzing received audio samples and generating audio models" refers to a method by which a server analyzes received audio data and constructs individual audio models based on that data.

[0553] "Method for registering a generated voice model as an NFT on the blockchain" refers to a method for recording the metadata of a generated voice model on the blockchain as an NFT and digitally proving the rights to it.

[0554] "Means for receiving real-time audio data and performing audio conversion by applying a generated audio model" refers to means for receiving audio data provided by the user in real time and performing conversion processing by applying an existing audio model.

[0555] "A means of performing speech conversion processing using GPU resources on the cloud" refers to a method of efficiently distributing the computational load of speech conversion processing by utilizing GPU resources located in a cloud environment.

[0556] "A means of sending the converted audio data back to the user in real time" refers to a means of immediately delivering the converted audio data to the user's device.

[0557] "A means of enabling a virtual character to perform in real time using a user's voice by utilizing a generated voice model" refers to a method of enabling a virtual character to play back the user's voice in real time using live streaming or recording, by leveraging a generated voice model.

[0558] This invention relates to a system that generates individual voice models using voice samples provided by the user and performs real-time voice conversion using these models. The overall system configuration and specific embodiments are described below.

[0559] Server-side processing

[0560] User registration and receiving audio samples

[0561] The server accepts users creating an account and recording and submitting a voice sample. The user's account information and voice sample are stored in the server's database, and a unique user ID is generated. This process identifies each individual user.

[0562] Voice model generation and NFT registration

[0563] The received audio samples are analyzed using AI technology to generate individual voice models. These voice models reproduce the user's specific voice characteristics and speaking style. Furthermore, the metadata of these generated voice models is registered on the blockchain as NFTs, digitally proving the copyright and usage rights of the voice models. Smart contract technology is used to ensure the secure management of these rights.

[0564] Real-time speech conversion

[0565] When a user utilizes the real-time speech conversion function, the server receives streaming audio data from the user and applies a generated speech model to perform the conversion. This process leverages GPU resources in the cloud to achieve high-speed and efficient speech conversion. The converted audio data is returned to the user in real time.

[0566] Terminal-side processing

[0567] Recording and sending audio samples

[0568] The terminal provides an interface for the user to record voice samples. The recorded voice samples are sent to a server and used for analysis and voice model generation.

[0569] Real-time speech conversion operation

[0570] When a user utilizes the real-time voice conversion function, the device acquires voice data via the microphone and streams it to the server. After the voice conversion process on the server is complete, the converted audio is played back in real time from the device.

[0571] User-side operations

[0572] User registration and provision of audio samples

[0573] Users create an account using the application, record a voice sample, and send it to the system. At this stage, a voice model is generated and registered on the blockchain as an NFT.

[0574] Using real-time speech conversion

[0575] When a user utilizes the real-time voice conversion function, the device sends audio to the server in real time and receives the converted audio in real time. This allows users to apply their own voice to virtual characters and perform with them.

[0576] Program Processing Description

[0577] This system uses Python, the requests library, and the sounddevice library to record, send, and perform real-time speech conversion of audio samples. On the server side, cloud-based GPU resources are used to process the speech conversion at high speed. The generated speech models are then registered on the blockchain as NFTs using smart contracts.

[0578] Specific example

[0579] For example, when a user uses a virtual performance application, the following process takes place:

[0580] 1. The user creates an account, records an audio sample according to the instructions, and sends it to the server.

[0581] 2. The server receives the voice sample, analyzes it, and generates a voice model.

[0582] 3. The voice model will be registered on the blockchain as an NFT.

[0583] 4. Users utilize a real-time voice conversion function to apply their own voice to a virtual character and perform.

[0584] Example of a prompt:

[0585] This program records a voice sample provided by the user, sends it to an API server to generate a voice model, and further includes the process of converting and playing back the voice in real time. Below is an example of the code that records the user's voice and sends the sample to the server.

[0586] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0587] Step 1:

[0588] User registration and recording and submission of voice samples.

[0589] The user accesses the application using their device and creates an account. Once the user enters and submits their account information (username, password, etc.), the server receives it and stores it in its database. Next, the user records a voice sample using their device's microphone and sends the data to the server. The server receives this voice sample and temporarily stores it for analysis.

[0590] Input: User account information and voice sample

[0591] Output: User ID and voice sample data

[0592] Step 2:

[0593] Voice model generation and NFT registration

[0594] The server analyzes the received audio sample and generates a voice model using an AI model based on that data. This voice model reproduces the user's specific voice quality and speaking style. The metadata of the generated voice model is registered on the blockchain as an NFT, where digital copyright management takes place.

[0595] Input: Audio sample data

[0596] Output: Voice model registered as an NFT

[0597] Step 3:

[0598] Preparing for real-time speech conversion

[0599] The user selects the real-time speech conversion function using their device. The device uses its microphone to acquire voice data in real time and prepares to send it to the server. At this time, the server prepares to use the already generated voice model.

[0600] Input: User's real-time voice data, existing voice model

[0601] Output: Real-time streaming start notification

[0602] Step 4:

[0603] Receiving and converting real-time audio data

[0604] The terminal sends real-time audio data from the user to the server in streaming format. The server receives this audio data and applies a generative AI model to perform real-time speech conversion. This process utilizes GPU resources in the cloud to achieve high-speed and efficient speech conversion.

[0605] Input: Real-time audio data sent by the user

[0606] Output: Converted audio data

[0607] Step 5:

[0608] Return and playback of the converted audio data

[0609] The server immediately returns the converted audio data to the user's terminal. The terminal plays this received data in real time, allowing the user to instantly check the converted audio. This process enables the virtual character to perform in real time using the user's voice.

[0610] Input: Converted audio data

[0611] Output: Audio played back in real time

[0612] Through these steps, individual voice models are generated based on voice samples provided by the user, and these models are used to achieve real-time voice conversion. Furthermore, the copyright of the generated voice models can be registered as NFTs on the blockchain, allowing for digital rights management.

[0613] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0614] The system of this invention creates individual voice models using a voice generation AI and an emotion engine, manages their rights using NFTs, and performs real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described in detail below.

[0615] Server-side processing

[0616] User registration and receiving audio samples

[0617] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[0618] Voice model generation, registration, and sentiment analysis

[0619] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific characteristics and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[0620] Real-time speech conversion and emotion recognition

[0621] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results from the sentiment engine to perform the conversion. In this process, GPU resources in the cloud are utilized to efficiently execute the complex conversion process. The converted audio data is returned to the user in real time.

[0622] Terminal-side processing

[0623] Recording and sending audio samples

[0624] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[0625] Real-time speech conversion operation

[0626] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[0627] User-side operations

[0628] User registration and provision of audio samples

[0629] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[0630] Using real-time speech conversion

[0631] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[0632] Specific example

[0633] When a user records an audio sample and sends it to the server

[0634] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates a voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[0635] When a user uses real-time speech conversion

[0636] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[0637] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[0638] The following describes the processing flow.

[0639] Server-side processing

[0640] Step 1:

[0641] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID.

[0642] Step 2:

[0643] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format. After saving, a confirmation message is sent to the user.

[0644] Step 3:

[0645] The server analyzes the received audio sample and generates an audio model using an AI model. This analysis process extracts feature elements from the audio sample.

[0646] Step 4:

[0647] The server uses an emotion engine to analyze the user's emotions from the voice sample. The results of the emotion analysis are reflected in the voice model.

[0648] Step 5:

[0649] The server describes the metadata of the generated voice model and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[0650] Step 6:

[0651] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[0652] Step 7:

[0653] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion process is performed by applying the speech model and sentiment analysis results, generating audio data at high speed.

[0654] Step 8:

[0655] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[0656] Terminal-side processing

[0657] Step 1:

[0658] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[0659] Step 2:

[0660] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[0661] Step 3:

[0662] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[0663] Step 4:

[0664] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[0665] Step 5:

[0666] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[0667] User-side operations

[0668] Step 1:

[0669] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[0670] Step 2:

[0671] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[0672] Step 3:

[0673] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[0674] Step 4:

[0675] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[0676] (Example 2)

[0677] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0678] In recent years, speech generation technology has advanced dramatically, but there is a need for a method to efficiently generate speech models that reflect the individual voice quality and emotions of each user, and to clearly manage their ownership and usage rights. Furthermore, real-time, high-quality speech conversion requires significant computing resources, necessitating systems that can process this efficiently.

[0679] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to extract features and generate a voice model, means for analyzing the user's emotions from the voice data and reflecting the results in the voice model, means for registering the metadata of the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model and emotion analysis results, means for executing high-load voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This enables the efficient generation of voice models that reflect the user's individual voice quality and emotions, and high-quality voice conversion in real time.

[0680] A "user" is a person or organization that uses the system and is the entity that performs operations such as providing voice samples or transmitting voice data in real time.

[0681] A "voice sample" is audio data that a user records and provides to the system, and is used to understand the user's voice quality and speaking style.

[0682] "Features" are acoustic characteristics extracted from audio data and are used to generate audio models. Examples include Mel-frequency cepstrum coefficients (MFCCs).

[0683] A "voice model" is a model generated by analyzing received voice samples and based on their features, representing a digital voice that reflects the user's voice quality and emotions.

[0684] "Emotion analysis" is the process of estimating a user's emotional state from voice data, and is performed using an emotion engine.

[0685] "NFT" stands for Non-Fungible Token, a technology used to prove ownership and usage rights of digital data on the blockchain.

[0686] Blockchain is a system for securely and decentralizedly managing digital data, and it is a technology that records data in a series of "blocks" and links them together in a chain.

[0687] "Real-time audio data" refers to audio data provided by the user in real time through a microphone, which is then sent to a server and analyzed and converted in real time.

[0688] "GPU resources" refer to the computing power of a graphics processing unit and are resources used to efficiently perform computationally intensive tasks.

[0689] "Voice conversion" is the process of analyzing received real-time voice data and converting it by applying the generated voice model and sentiment analysis results.

[0690] "Metadata" refers to supplementary materials containing information about the voice model, and is data registered on the blockchain as an NFT.

[0691] The system of the present invention combines speech generation technology and emotion analysis technology, registers them as NFTs, and performs speech conversion in real time. The overall system configuration and specific embodiments are described in detail below.

[0692] Server-side processing

[0693] The server performs the following main tasks:

[0694] 1. User registration and receipt of audio samples

[0695] The server receives account creation requests from users and stores the entered information in a database (e.g., MySQL or PostgreSQL). It generates a user ID, receives a voice sample, and temporarily stores it in cloud storage (e.g., Amazon S3 or Google Cloud Storage).

[0696] 2. Generation of voice models and sentiment analysis

[0697] The server uses a speech analysis library (e.g., Librosa or PyDub) to analyze the received audio sample and extract features. Based on the extracted features, it uses a generative AI model (e.g., TensorFlow or PyTorch) to generate an audio model. An emotion engine (e.g., EmotionAI) analyzes the emotions from the audio data and incorporates the results into the audio model.

[0698] 3. Register the voice model as an NFT.

[0699] The server describes the metadata of the generated voice model and registers it on a blockchain (e.g., Ethereum or Solana) using a smart contract. This digitally proves ownership and usage rights of the voice model.

[0700] 4. Real-time speech conversion and emotion recognition

[0701] The server receives real-time audio data from the user, analyzes the data, applies a generated audio model and sentiment analysis results, and performs speech conversion. This process utilizes GPU resources in the cloud (e.g., NVIDIA GPU Cloud). The converted audio data is returned to the user in real time.

[0702] Terminal-side processing

[0703] The terminal performs the following main processes:

[0704] 1. Record and send audio samples.

[0705] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, which is then sent to the server. The user records a voice sample, which is temporarily stored and sent to the server once the recording is complete.

[0706] 2. Operating Real-Time Speech Conversion

[0707] The device has a real-time voice conversion function for the user. When the user provides voice in real time, the voice data is streamed to the server. After the conversion process is performed on the server side, the converted voice data is played back on the device.

[0708] User-side operations

[0709] The user performs the following main operations:

[0710] 1. User registration and provision of audio samples

[0711] Users create an account through the application and proceed to a page for providing a voice sample. They register their voice by recording a voice sample and submitting it to the system.

[0712] 2. Use of real-time speech conversion

[0713] The user selects the real-time voice conversion function within the application and provides voice in real time to the microphone. The voice data is processed on the server side, the emotional state is recognized, and the appropriately adjusted voice is played back in real time from the device.

[0714] Specific example

[0715] An example of the specific processing procedure is shown below.

[0716] When a user records an audio sample and sends it to the server

[0717] 1. The user enters their name, email address, password, etc. on the account creation screen and presses the submit button.

[0718] 2. The server saves the received information to the database and generates a unique user ID.

[0719] 3. The user follows the application's instructions and records a 60-second audio sample via the microphone.

[0720] 4. After recording is complete, the device uploads the audio file to the server.

[0721] 5. The server analyzes the audio data and generates an audio model.

[0722] 6. The emotion engine analyzes the audio data and incorporates the emotional information into the audio model.

[0723] 7. The server describes the metadata of the voice model and registers it on the blockchain.

[0724] 8. A completion notification is sent to the user.

[0725] When a user uses real-time speech conversion

[0726] 1. The user launches the application and selects the real-time speech conversion function.

[0727] 2. The device captures the user's real-time voice through the microphone and streams it to the server.

[0728] 3. The server analyzes the received audio data and performs speech conversion by applying the audio model and sentiment analysis results.

[0729] 4. The converted audio data is efficiently processed using GPU resources in the cloud.

[0730] 5. The converted audio data is sent back to the device in real time and played back through the speaker or headphones.

[0731] Examples of input prompt messages

[0732] "Please register a new account. Enter the required information and record and submit a 60-second voice sample."

[0733] "Please begin real-time speech conversion. When you speak into the microphone, your voice will be automatically sent to the server."

[0734] This system utilizes generative AI models and an emotion engine to consistently support the entire process, from generating voice models to rights management, emotion analysis, and real-time speech conversion. Furthermore, it leverages cloud-based GPU resources to efficiently perform advanced processing.

[0735] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0736] Step 1:

[0737] The user enters the required information on the account creation screen and presses the submit button. This information includes name, email address, and password. The server receives this information and stores it in the database. Based on the information stored in the database, the server generates a unique user ID and stores this user ID in the database. The input data is the user's registration information, and the output is the user ID stored in the database.

[0738] Step 2:

[0739] The user proceeds to a screen to record a voice sample. The user presses the record button to record a voice sample through the microphone. Once recording is complete, the device temporarily saves the voice sample data. The input is the user's voice data, and the output is the temporarily saved voice data.

[0740] Step 3:

[0741] The device sends temporarily stored audio data to the server. At the same time, metadata such as the audio file format (e.g., WAV format) and sampling rate is also sent. The server saves the received audio sample to cloud storage. The data received by the server is the audio sample, and the output is the audio data stored in cloud storage.

[0742] Step 4:

[0743] The server analyzes the audio sample and extracts features. Specifically, it uses an audio analysis library (e.g., Librosa or PyDub) to extract acoustic features. The extracted features include MFCC (Mel-frequency cepstrum coefficients) and zero crossover ratio. The input data is an audio sample, and the output is the extracted features.

[0744] Step 5:

[0745] The server generates a speech model using a generative AI model based on the extracted features. The speech model is constructed using AI model libraries such as TensorFlow or PyTorch. The input data consists of features, and the output is the speech model.

[0746] Step 6:

[0747] The server analyzes the user's emotions from the audio data. An emotion engine (e.g., EmotionAI) is used to estimate the emotional state within the audio data. The analysis results are output as emotion labels such as joy, sadness, and anger. The input is audio data, and the output is emotion labels.

[0748] Step 7:

[0749] The server incorporates the emotion analysis results into the speech model. By incorporating emotion labels into the speech model, it becomes possible to generate speech that reflects the user's emotional state. The input data consists of the speech model and emotion labels, and the output is the speech model that reflects the emotions.

[0750] Step 8:

[0751] The server describes the metadata of the generated voice model and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model. The input is the metadata of the voice model, and the output is an NFT registered on the blockchain.

[0752] Step 9:

[0753] When a user uses the real-time speech conversion function, the device captures audio data in real time and streams it to the server. The input is real-time audio data, and the output is streamed audio data.

[0754] Step 10:

[0755] The server analyzes real-time audio data it receives and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, high-load processing is efficiently executed using GPU resources in the cloud. The input data consists of real-time audio data and a speech model, while the output is the converted audio data.

[0756] Step 11:

[0757] The server sends the converted audio data back to the user in real time. The terminal plays the received converted audio data through speakers or headphones. The input is the converted audio data, and the output is the played audio.

[0758] This ensures that specific data processing and calculations are performed at each processing step, ultimately resulting in a system that allows users to smoothly utilize voice conversion.

[0759] (Application Example 2)

[0760] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0761] In traditional online shopping, it was difficult to provide personalized voice models or emotionally resonant voices for virtual assistants when they interacted with users. This resulted in a mechanical user experience, leading to decreased emotional satisfaction and user engagement. Furthermore, there was a lack of efficient methods for handling the high-load processing required for real-time voice conversion.

[0762] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0763] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate individual voice models, means for registering the generated voice models as NFTs on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice models and sentiment analysis, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time and appropriately adjusting the voice based on the user's emotional state. This enables interactive and emotional responses by a personalized voice assistant in virtual stores, improving the user experience. Furthermore, it allows for the efficient distribution and execution of high-load cloud-based processing.

[0764] "Means for receiving voice samples from users" refers to devices or software that provide an interface for users to record their own voice samples and send them to the system.

[0765] "Means for analyzing received audio samples and generating individual audio models" refers to devices or algorithms that extract feature quantities from audio samples and generate user-specific audio models.

[0766] "Methods for registering generated voice models as NFTs on the blockchain" refers to devices or software that generate metadata for a voice model as an NFT and register it on the blockchain using smart contracts.

[0767] "Means for receiving real-time voice data and performing voice conversion by applying a generated voice model and sentiment analysis" refers to a device or software that receives real-time voice data from a user and converts it using a user-specific voice model and sentiment analysis engine.

[0768] "Means of performing speech conversion processing using cloud-based GPU resources" refers to devices or software that efficiently perform the high-load processing of real-time speech conversion by utilizing cloud-based high-performance GPU resources.

[0769] "Means for returning converted audio data to the user in real time and appropriately adjusting the audio based on the user's emotional state" refers to a device or software that immediately returns converted audio data to the user and further appropriately adjusts the audio based on the user's emotional state.

[0770] This invention realizes a system in which a virtual assistant with an individualized voice model guides users while they enjoy shopping in a virtual store. The overall system configuration and specific embodiments are described in detail below.

[0771] Server-side processing

[0772] User registration and receiving audio samples

[0773] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[0774] Voice model generation, registration, and sentiment analysis

[0775] The server analyzes the received audio sample and generates an individual audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion analysis engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific features and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[0776] Real-time speech conversion and emotion recognition

[0777] Real-time speech-to-text conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results to perform the conversion. During this process, GPU resources in the cloud (e.g., Google Cloud Speech-to-Text) are utilized to efficiently execute complex conversion processes. The converted audio data is then returned to the user in real time.

[0778] Terminal-side processing

[0779] Recording and sending audio samples

[0780] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[0781] Real-time speech conversion operation

[0782] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[0783] User-side operations

[0784] User registration and provision of audio samples

[0785] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[0786] Using real-time speech conversion

[0787] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[0788] Specific example

[0789] When a user records an audio sample and sends it to the server

[0790] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates an individual voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[0791] When a user uses real-time speech conversion

[0792] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[0793] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[0794] Examples of prompt statements

[0795] For example, if a user asks the virtual assistant, "Tell me more about this product," the following prompt will be generated.

[0796] The user asked, "Tell me more about this product."

[0797] Emotion analysis result: Very interesting

[0798] Generated audio: An audio that enthusiastically explains detailed information.

[0799] This prompt allows for the generation of appropriate messages in real time, based on sentiment analysis results, and provides them to the user.

[0800] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0801] Step 1:

[0802] To register an account, users access the account creation screen through the application, enter the required information, and submit it.

[0803] Input: User information (name, email address, password, etc.)

[0804] Output: Unique User ID

[0805] Specific operation: The terminal sends the entered user information to the server, which stores this information in a database and generates a unique user ID.

[0806] Step 2:

[0807] The user records an audio sample. Once recording is complete, the device sends the audio sample data to the server.

[0808] Input: User voice sample

[0809] Output: Temporarily saved audio sample

[0810] Specific operation: The terminal records an audio sample, temporarily stores it, and then provides an interface for sending it to the server. The server receives the audio sample and temporarily stores it.

[0811] Step 3:

[0812] The server analyzes the received audio samples and generates individual audio models. During this process, it uses AI models to extract feature vectors from the audio samples.

[0813] Input: Audio sample

[0814] Output: Individual speech models

[0815] Specific operation: The server uses an AI library for speech generation to analyze speech samples, extract features, and then generate individual speech models.

[0816] Step 4:

[0817] The metadata of the generated voice model is registered on the blockchain as an NFT.

[0818] Input: Voice model

[0819] Output: NFT-registered voice models

[0820] Specific operation: The server generates voice model information as an NFT and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model.

[0821] Step 5:

[0822] To utilize the real-time voice conversion function, the user provides voice data via the microphone. The device then streams this real-time voice data to the server.

[0823] Input: Real-time audio data

[0824] Output: Streamed audio data

[0825] Specific operation: The device captures audio data in real time and streams it to the server.

[0826] Step 6:

[0827] The server analyzes the received real-time audio data and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, it utilizes GPU resources in the cloud to perform high-speed conversion.

[0828] Input: Real-time audio data, audio model, sentiment analysis results

[0829] Output: Converted audio data

[0830] Specific operation: The server uses cloud-based GPU resources to analyze and convert real-time audio data, and then applies the sentiment analysis results to adjust the audio.

[0831] Step 7:

[0832] The converted audio data is sent back to the user in real time, and the audio is appropriately adjusted based on the user's emotional state.

[0833] Input: Converted audio data, user's emotional state

[0834] Output: Real-time adjusted converted audio

[0835] Specific operation: The server immediately returns the converted audio data to the user and adjusts the audio appropriately according to the user's emotional state. The terminal receives this data and plays it back in real time through speakers or headphones.

[0836] In this way, by sequentially executing the specific processes performed at each step, the virtual assistant can provide a personalized voice model and an interactive user experience based on sentiment analysis.

[0837] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0838] The data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0839] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0840] [Third Embodiment]

[0841] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0842] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0843] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0844] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0845] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0846] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0847] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0848] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0849] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0850] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0851] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0852] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0853] The system of the present invention can create individual voice models using a voice generation AI, manage their rights using NFTs, and perform real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described below.

[0854] Server-side processing

[0855] User registration and receiving audio samples

[0856] The server first receives a user registration request. Once the user creates an account, the server stores this information in a database and generates a unique user ID. It then receives and temporarily stores a voice sample submitted by the user. The voice sample is used to understand the user's specific voice characteristics and speaking style.

[0857] Voice model generation and registration

[0858] The server analyzes the received audio sample and generates an audio model using AI technology. The generated audio model has user-specific characteristics and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[0859] Real-time speech conversion

[0860] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies a generated speech model to perform the conversion. During this process, GPU resources in the cloud are utilized to efficiently execute complex conversion operations. The converted audio data is then returned to the user in real time.

[0861] Terminal-side processing

[0862] Recording and sending audio samples

[0863] The user records an audio sample using the microphone on their smartphone or computer. Once the recording is complete, the device provides an interface for sending this data to the server.

[0864] Real-time speech conversion operation

[0865] When a user utilizes the real-time speech conversion function, the device streams audio data to the server in real time. After the server performs the conversion process, the device plays the converted audio in real time.

[0866] User-side operations

[0867] User registration and provision of audio samples

[0868] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[0869] Using real-time speech conversion

[0870] The user selects a conversion function and provides the audio in real time. The converted audio, processed on the server side, is played back on the device in real time and can be used directly by the user.

[0871] Specific example

[0872] When a user records an audio sample and sends it to the server

[0873] The user creates an account and records a 60-second audio sample following the application's instructions. Once recording is complete, the device sends the audio data to the server. The server analyzes the received audio sample and generates an audio model. The metadata of the audio model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[0874] When a user uses real-time speech conversion

[0875] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a generated speech model. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is sent back to the device in real time for the user to use.

[0876] This system provides comprehensive support for the entire process, from voice model generation to rights management and real-time voice conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[0877] The following describes the processing flow.

[0878] Server-side processing

[0879] Step 1:

[0880] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID.

[0881] Step 2:

[0882] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format.

[0883] Step 3:

[0884] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample, which are then input into the AI ​​model.

[0885] Step 4:

[0886] The server describes the generated voice model information as metadata and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[0887] Step 5:

[0888] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[0889] Step 6:

[0890] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion is performed by applying a speech model, generating audio data at high speed.

[0891] Step 7:

[0892] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[0893] Terminal-side processing

[0894] Step 1:

[0895] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[0896] Step 2:

[0897] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[0898] Step 3:

[0899] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[0900] Step 4:

[0901] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[0902] Step 5:

[0903] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[0904] User-side operations

[0905] Step 1:

[0906] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[0907] Step 2:

[0908] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[0909] Step 3:

[0910] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[0911] Step 4:

[0912] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[0913] (Example 1)

[0914] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0915] Existing voice conversion systems lack sufficient security and reliability in terms of the individuality and copyright management of voice data, and they have difficulty responding quickly to high-load voice conversion processing. Furthermore, there is a need for technology that can efficiently perform real-time voice conversion while protecting the copyright of voice data provided by users.

[0916] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0917] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model on a blockchain as a digital proof, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using remote computing resources, and means for returning the voice-converted data to the user in real time. This makes it possible to efficiently perform high-load voice conversion processing while ensuring the individuality of the voice data and copyright protection.

[0918] "Users" refer to individuals or companies that utilize the system, including those who provide voice samples or use real-time voice conversion.

[0919] A "voice sample" refers to data consisting of a user's own recorded voice, and is the basic data used to generate a voice model.

[0920] A "voice model" is a digital acoustic profile generated from analyzed voice samples, and refers to a model that possesses specific voice characteristics or speaking style traits.

[0921] "Digital proof means" refers to a technology that registers the metadata of a generated voice model on a distributed ledger technology (blockchain) to digitally prove ownership and usage rights.

[0922] "Blockchain" refers to a distributed ledger technology composed of multiple nodes, designed to ensure data transparency and prevent tampering.

[0923] "Real-time audio data" refers to audio data provided by users in real time, which is immediately converted into speech.

[0924] "Remote computing resources" refer to high-performance computing resources (e.g., GPUs) located in the cloud, which are used to efficiently distribute and execute high-load computational processes such as speech conversion.

[0925] "Speech conversion processing" refers to the process of applying a generated speech model to received real-time speech data and converting it into different speech characteristics.

[0926] The system of the present invention utilizes a speech generation AI to create individual speech models, manages their rights as digital proof-of-services (NFTs), and performs real-time speech conversion using cloud-based remote computing resources. This embodiment includes the processes of receiving speech samples, generating and registering speech models, and real-time speech conversion.

[0927] Server-side processing

[0928] The server first receives a voice sample from the user. The received voice sample is temporarily stored and analyzed in detail using AI technology. Deep learning frameworks such as TensorFlow and PyTorch are used for this analysis. As a result of the analysis, a user-specific voice model is generated. This voice model is described as metadata and registered on the blockchain as a means of digital proof. Distributed ledger technology is used as the blockchain platform. This digitally proves ownership and usage rights of the voice model.

[0929] Next, in the real-time speech conversion process, the user sends voice data to the server in real time via their device. The server analyzes the received voice data and performs speech conversion by applying a pre-generated voice model. During this process, a high-performance GPU in the cloud (such as NVIDIA Tesla V100) is used to efficiently perform the speech conversion. The converted voice data is then returned to the user in real time.

[0930] Terminal-side processing

[0931] The device provides an interface for recording user-provided audio samples and sending them to the server. When the user records an audio sample and presses the send button, this audio data is sent to the server. Furthermore, when the user utilizes the real-time voice conversion function, the device streams the audio data in real time via the microphone and plays the converted audio in real time.

[0932] User-side operations

[0933] Users create an account through the application and proceed to a page for providing a voice sample. By recording and submitting a voice sample according to the instructions, they register their voice with the system. They can also select the real-time voice conversion function and provide their voice in real time. The converted voice, processed on the server side, is played back on the device in real time and can be used directly by the user.

[0934] Specific example

[0935] 1. Prompt for user registration and voice sample provision

[0936] Launch the application and create an account. Then, follow the instructions to record a 60-second voice sample.

[0937] 2. Prompts for real-time speech conversion

[0938] To use the real-time speech conversion feature, prepare your microphone and press the conversion button in the application. Your speech will be converted in real time.

[0939] As described above, the system of the present invention provides a series of processes in which a user provides a voice sample, a voice model is generated and registered based on it, and voice conversion is performed in real time. This makes it possible to achieve efficient voice conversion while ensuring the security and reliability of voice data.

[0940] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0941] Step 1:

[0942] The user enters information such as their name, email address, and password on the account creation screen and presses the submit button. The entered information is sent from the device to the server. The server stores the received information in a database and generates a unique user ID. This is to manage the user's basic information, and the database includes the user ID, name, email address, etc. The user ID generated by the server is output.

[0943] Step 2:

[0944] The user proceeds to the audio sample provision page and records an audio sample using their device's microphone. Once recording is complete, the user presses the send button, and the audio data is sent from the device to the server. The server receives the audio data and temporarily stores it. This audio data is the basis for analyzing the user's voice quality and speaking style. An output confirms that the received audio data has been saved to the server.

[0945] Step 3:

[0946] The server analyzes the stored audio samples and generates an audio model using AI technology. Specifically, it extracts audio features using deep learning frameworks such as TensorFlow and PyTorch to generate an acoustic model. This model has user-specific audio characteristics. The generated audio model is output as a result of the analysis.

[0947] Step 4:

[0948] The server creates metadata for the generated voice model and registers it on the blockchain as a digital proof of value (NFT). A smart contract is used in this process, and the voice model's metadata is recorded on the blockchain platform (e.g., Ethereum or Hyperledger). This digitally proves ownership and usage rights of the voice model. The generated voice model's metadata and registration completion information are output.

[0949] Step 5:

[0950] If a user desires real-time speech conversion, they launch the application on their device and provide real-time audio data through the microphone. The device streams the real-time audio data to the server. The server receives this audio data and applies a pre-generated speech model to perform real-time speech conversion. This process is efficiently performed using a high-performance GPU in the cloud (e.g., NVIDIA Tesla V100). The converted audio data is output and immediately returned to the user.

[0951] Step 6:

[0952] Once the converted audio data is sent back to the device, the device plays this data in real time. The user can immediately listen to the converted audio. This process continues as long as the user continues to provide audio in real time. The output converted audio data is played back to the user in real time.

[0953] (Application Example 1)

[0954] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0955] This invention relates to a system that uses a speech generation AI to create individual speech models and converts speech in real time based on those models. Conventional technologies have problems with inefficient processing, requiring significant computational resources and time for creating speech models and real-time speech conversion. Furthermore, the rights management of speech models is ambiguous, creating risks of theft and misuse. In addition, it has been difficult for users to control virtual characters with their own voices. This invention aims to solve these problems and provide a highly efficient and secure speech conversion system.

[0956] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0957] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This ensures reliable rights management of the voice model and enables high-speed and highly efficient real-time voice conversion. Furthermore, the generated voice model enables a virtual character to perform in real time using the user's voice.

[0958] "Means for receiving audio samples from users" refers to means for recording audio data provided by users and sending it to the server.

[0959] "Means for analyzing received audio samples and generating audio models" refers to a method by which a server analyzes received audio data and constructs individual audio models based on that data.

[0960] "Method for registering a generated voice model as an NFT on the blockchain" refers to a method for recording the metadata of a generated voice model on the blockchain as an NFT and digitally proving the rights to it.

[0961] "Means for receiving real-time audio data and performing audio conversion by applying a generated audio model" refers to means for receiving audio data provided by the user in real time and performing conversion processing by applying an existing audio model.

[0962] "A means of performing speech conversion processing using GPU resources on the cloud" refers to a method of efficiently distributing the computational load of speech conversion processing by utilizing GPU resources located in a cloud environment.

[0963] "A means of sending the converted audio data back to the user in real time" refers to a means of immediately delivering the converted audio data to the user's device.

[0964] "A means of enabling a virtual character to perform in real time using a user's voice by utilizing a generated voice model" refers to a method of enabling a virtual character to play back the user's voice in real time using live streaming or recording, by leveraging a generated voice model.

[0965] This invention relates to a system that generates individual voice models using voice samples provided by the user and performs real-time voice conversion using these models. The overall system configuration and specific embodiments are described below.

[0966] Server-side processing

[0967] User registration and receiving audio samples

[0968] The server accepts users creating an account and recording and submitting a voice sample. The user's account information and voice sample are stored in the server's database, and a unique user ID is generated. This process identifies each individual user.

[0969] Voice model generation and NFT registration

[0970] The received audio samples are analyzed using AI technology to generate individual voice models. These voice models reproduce the user's specific voice characteristics and speaking style. Furthermore, the metadata of these generated voice models is registered on the blockchain as NFTs, digitally proving the copyright and usage rights of the voice models. Smart contract technology is used to ensure the secure management of these rights.

[0971] Real-time speech conversion

[0972] When a user utilizes the real-time speech conversion function, the server receives streaming audio data from the user and applies a generated speech model to perform the conversion. This process leverages GPU resources in the cloud to achieve high-speed and efficient speech conversion. The converted audio data is returned to the user in real time.

[0973] Terminal-side processing

[0974] Recording and sending audio samples

[0975] The terminal provides an interface for the user to record voice samples. The recorded voice samples are sent to a server and used for analysis and voice model generation.

[0976] Real-time speech conversion operation

[0977] When a user utilizes the real-time voice conversion function, the device acquires voice data via the microphone and streams it to the server. After the voice conversion process on the server is complete, the converted audio is played back in real time from the device.

[0978] User-side operations

[0979] User registration and provision of audio samples

[0980] Users create an account using the application, record a voice sample, and send it to the system. At this stage, a voice model is generated and registered on the blockchain as an NFT.

[0981] Using real-time speech conversion

[0982] When a user utilizes the real-time voice conversion function, the device sends audio to the server in real time and receives the converted audio in real time. This allows users to apply their own voice to virtual characters and perform with them.

[0983] Program Processing Description

[0984] This system uses Python, the requests library, and the sounddevice library to record, send, and perform real-time speech conversion of audio samples. On the server side, cloud-based GPU resources are used to process the speech conversion at high speed. The generated speech models are then registered on the blockchain as NFTs using smart contracts.

[0985] Specific example

[0986] For example, when a user uses a virtual performance application, the following process takes place:

[0987] 1. The user creates an account, records an audio sample according to the instructions, and sends it to the server.

[0988] 2. The server receives the voice sample, analyzes it, and generates a voice model.

[0989] 3. The voice model will be registered on the blockchain as an NFT.

[0990] 4. Users utilize a real-time voice conversion function to apply their own voice to a virtual character and perform.

[0991] Example of a prompt:

[0992] This program records a voice sample provided by the user, sends it to an API server to generate a voice model, and further includes the process of converting and playing back the voice in real time. Below is an example of the code that records the user's voice and sends the sample to the server.

[0993] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0994] Step 1:

[0995] User registration and recording and submission of voice samples.

[0996] The user accesses the application using their device and creates an account. Once the user enters and submits their account information (username, password, etc.), the server receives it and stores it in its database. Next, the user records a voice sample using their device's microphone and sends the data to the server. The server receives this voice sample and temporarily stores it for analysis.

[0997] Input: User account information and voice sample

[0998] Output: User ID and voice sample data

[0999] Step 2:

[1000] Voice model generation and NFT registration

[1001] The server analyzes the received audio sample and generates a voice model using an AI model based on that data. This voice model reproduces the user's specific voice quality and speaking style. The metadata of the generated voice model is registered on the blockchain as an NFT, where digital copyright management takes place.

[1002] Input: Audio sample data

[1003] Output: Voice model registered as an NFT

[1004] Step 3:

[1005] Preparing for real-time speech conversion

[1006] The user selects the real-time speech conversion function using their device. The device uses its microphone to acquire voice data in real time and prepares to send it to the server. At this time, the server prepares to use the already generated voice model.

[1007] Input: User's real-time voice data, existing voice model

[1008] Output: Real-time streaming start notification

[1009] Step 4:

[1010] Receiving and converting real-time audio data

[1011] The terminal sends real-time audio data from the user to the server in streaming format. The server receives this audio data and applies a generative AI model to perform real-time speech conversion. This process utilizes GPU resources in the cloud to achieve high-speed and efficient speech conversion.

[1012] Input: Real-time audio data sent by the user

[1013] Output: Converted audio data

[1014] Step 5:

[1015] Return and playback of the converted audio data

[1016] The server immediately returns the converted audio data to the user's terminal. The terminal plays this received data in real time, allowing the user to instantly check the converted audio. This process enables the virtual character to perform in real time using the user's voice.

[1017] Input: Converted audio data

[1018] Output: Audio played back in real time

[1019] Through these steps, individual voice models are generated based on voice samples provided by the user, and these models are used to achieve real-time voice conversion. Furthermore, the copyright of the generated voice models can be registered as NFTs on the blockchain, allowing for digital rights management.

[1020] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1021] The system of this invention creates individual voice models using a voice generation AI and an emotion engine, manages their rights using NFTs, and performs real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described in detail below.

[1022] Server-side processing

[1023] User registration and receiving audio samples

[1024] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[1025] Voice model generation, registration, and sentiment analysis

[1026] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific characteristics and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[1027] Real-time speech conversion and emotion recognition

[1028] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results from the sentiment engine to perform the conversion. In this process, GPU resources in the cloud are utilized to efficiently execute the complex conversion process. The converted audio data is returned to the user in real time.

[1029] Terminal-side processing

[1030] Recording and sending audio samples

[1031] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[1032] Real-time speech conversion operation

[1033] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[1034] User-side operations

[1035] User registration and provision of audio samples

[1036] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[1037] Using real-time speech conversion

[1038] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[1039] Specific example

[1040] When a user records an audio sample and sends it to the server

[1041] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates a voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[1042] When a user uses real-time speech conversion

[1043] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[1044] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[1045] The following describes the processing flow.

[1046] Server-side processing

[1047] Step 1:

[1048] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID.

[1049] Step 2:

[1050] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format. After saving, a confirmation message is sent to the user.

[1051] Step 3:

[1052] The server analyzes the received audio sample and generates an audio model using an AI model. This analysis process extracts feature elements from the audio sample.

[1053] Step 4:

[1054] The server uses an emotion engine to analyze the user's emotions from the voice sample. The results of the emotion analysis are reflected in the voice model.

[1055] Step 5:

[1056] The server describes the metadata of the generated voice model and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[1057] Step 6:

[1058] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[1059] Step 7:

[1060] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion process is performed by applying the speech model and sentiment analysis results, generating audio data at high speed.

[1061] Step 8:

[1062] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[1063] Terminal-side processing

[1064] Step 1:

[1065] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[1066] Step 2:

[1067] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[1068] Step 3:

[1069] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[1070] Step 4:

[1071] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[1072] Step 5:

[1073] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[1074] User-side operations

[1075] Step 1:

[1076] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[1077] Step 2:

[1078] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[1079] Step 3:

[1080] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[1081] Step 4:

[1082] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[1083] (Example 2)

[1084] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1085] In recent years, speech generation technology has advanced dramatically, but there is a need for a method to efficiently generate speech models that reflect the individual voice quality and emotions of each user, and to clearly manage their ownership and usage rights. Furthermore, real-time, high-quality speech conversion requires significant computing resources, necessitating systems that can process this efficiently.

[1086] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to extract features and generate a voice model, means for analyzing the user's emotions from the voice data and reflecting the results in the voice model, means for registering the metadata of the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model and emotion analysis results, means for executing high-load voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This enables the efficient generation of voice models that reflect the user's individual voice quality and emotions, and high-quality voice conversion in real time.

[1087] A "user" is a person or organization that uses the system and is the entity that performs operations such as providing voice samples or transmitting voice data in real time.

[1088] A "voice sample" is audio data that a user records and provides to the system, and is used to understand the user's voice quality and speaking style.

[1089] "Features" are acoustic characteristics extracted from audio data and are used to generate audio models. Examples include Mel-frequency cepstrum coefficients (MFCCs).

[1090] A "voice model" is a model generated by analyzing received voice samples and based on their features, representing a digital voice that reflects the user's voice quality and emotions.

[1091] "Emotion analysis" is the process of estimating a user's emotional state from voice data, and is performed using an emotion engine.

[1092] "NFT" stands for Non-Fungible Token, a technology used to prove ownership and usage rights of digital data on the blockchain.

[1093] Blockchain is a system for securely and decentralizedly managing digital data, and it is a technology that records data in a series of "blocks" and links them together in a chain.

[1094] "Real-time audio data" refers to audio data provided by the user in real time through a microphone, which is then sent to a server and analyzed and converted in real time.

[1095] "GPU resources" refer to the computing power of a graphics processing unit and are resources used to efficiently perform computationally intensive tasks.

[1096] "Voice conversion" is the process of analyzing received real-time voice data and converting it by applying the generated voice model and sentiment analysis results.

[1097] "Metadata" refers to supplementary materials containing information about the voice model, and is data registered on the blockchain as an NFT.

[1098] The system of the present invention combines speech generation technology and emotion analysis technology, registers them as NFTs, and performs speech conversion in real time. The overall system configuration and specific embodiments are described in detail below.

[1099] Server-side processing

[1100] The server performs the following main tasks:

[1101] 1. User registration and receipt of audio samples

[1102] The server receives account creation requests from users and stores the entered information in a database (e.g., MySQL or PostgreSQL). It generates a user ID, receives a voice sample, and temporarily stores it in cloud storage (e.g., Amazon S3 or Google Cloud Storage).

[1103] 2. Generation of voice models and sentiment analysis

[1104] The server uses a speech analysis library (e.g., Librosa or PyDub) to analyze the received audio sample and extract features. Based on the extracted features, it uses a generative AI model (e.g., TensorFlow or PyTorch) to generate an audio model. An emotion engine (e.g., EmotionAI) analyzes the emotions from the audio data and incorporates the results into the audio model.

[1105] 3. Register the voice model as an NFT.

[1106] The server describes the metadata of the generated voice model and registers it on a blockchain (e.g., Ethereum or Solana) using a smart contract. This digitally proves ownership and usage rights of the voice model.

[1107] 4. Real-time speech conversion and emotion recognition

[1108] The server receives real-time audio data from the user, analyzes the data, applies a generated audio model and sentiment analysis results, and performs speech conversion. This process utilizes GPU resources in the cloud (e.g., NVIDIA GPU Cloud). The converted audio data is returned to the user in real time.

[1109] Terminal-side processing

[1110] The terminal performs the following main processes:

[1111] 1. Record and send audio samples.

[1112] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, which is then sent to the server. The user records a voice sample, which is temporarily stored and sent to the server once the recording is complete.

[1113] 2. Operating Real-Time Speech Conversion

[1114] The device has a real-time voice conversion function for the user. When the user provides voice in real time, the voice data is streamed to the server. After the conversion process is performed on the server side, the converted voice data is played back on the device.

[1115] User-side operations

[1116] The user performs the following main operations:

[1117] 1. User registration and provision of audio samples

[1118] Users create an account through the application and proceed to a page for providing a voice sample. They register their voice by recording a voice sample and submitting it to the system.

[1119] 2. Use of real-time speech conversion

[1120] The user selects the real-time voice conversion function within the application and provides voice in real time to the microphone. The voice data is processed on the server side, the emotional state is recognized, and the appropriately adjusted voice is played back in real time from the device.

[1121] Specific example

[1122] An example of the specific processing procedure is shown below.

[1123] When a user records an audio sample and sends it to the server

[1124] 1. The user enters their name, email address, password, etc. on the account creation screen and presses the submit button.

[1125] 2. The server saves the received information to the database and generates a unique user ID.

[1126] 3. The user follows the application's instructions and records a 60-second audio sample via the microphone.

[1127] 4. After recording is complete, the device uploads the audio file to the server.

[1128] 5. The server analyzes the audio data and generates an audio model.

[1129] 6. The emotion engine analyzes the audio data and incorporates the emotional information into the audio model.

[1130] 7. The server describes the metadata of the voice model and registers it on the blockchain.

[1131] 8. A completion notification is sent to the user.

[1132] When a user uses real-time speech conversion

[1133] 1. The user launches the application and selects the real-time speech conversion function.

[1134] 2. The device captures the user's real-time voice through the microphone and streams it to the server.

[1135] 3. The server analyzes the received audio data and performs speech conversion by applying the audio model and sentiment analysis results.

[1136] 4. The converted audio data is efficiently processed using GPU resources in the cloud.

[1137] 5. The converted audio data is sent back to the device in real time and played back through the speaker or headphones.

[1138] Examples of input prompt messages

[1139] "Please register a new account. Enter the required information and record and submit a 60-second voice sample."

[1140] "Please begin real-time speech conversion. When you speak into the microphone, your voice will be automatically sent to the server."

[1141] This system utilizes generative AI models and an emotion engine to consistently support the entire process, from generating voice models to rights management, emotion analysis, and real-time speech conversion. Furthermore, it leverages cloud-based GPU resources to efficiently perform advanced processing.

[1142] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1143] Step 1:

[1144] The user enters the required information on the account creation screen and presses the submit button. This information includes name, email address, and password. The server receives this information and stores it in the database. Based on the information stored in the database, the server generates a unique user ID and stores this user ID in the database. The input data is the user's registration information, and the output is the user ID stored in the database.

[1145] Step 2:

[1146] The user proceeds to a screen to record a voice sample. The user presses the record button to record a voice sample through the microphone. Once recording is complete, the device temporarily saves the voice sample data. The input is the user's voice data, and the output is the temporarily saved voice data.

[1147] Step 3:

[1148] The device sends temporarily stored audio data to the server. At the same time, metadata such as the audio file format (e.g., WAV format) and sampling rate is also sent. The server saves the received audio sample to cloud storage. The data received by the server is the audio sample, and the output is the audio data stored in cloud storage.

[1149] Step 4:

[1150] The server analyzes the audio sample and extracts features. Specifically, it uses an audio analysis library (e.g., Librosa or PyDub) to extract acoustic features. The extracted features include MFCC (Mel-frequency cepstrum coefficients) and zero crossover ratio. The input data is an audio sample, and the output is the extracted features.

[1151] Step 5:

[1152] The server generates a speech model using a generative AI model based on the extracted features. The speech model is constructed using AI model libraries such as TensorFlow or PyTorch. The input data consists of features, and the output is the speech model.

[1153] Step 6:

[1154] The server analyzes the user's emotions from the audio data. An emotion engine (e.g., EmotionAI) is used to estimate the emotional state within the audio data. The analysis results are output as emotion labels such as joy, sadness, and anger. The input is audio data, and the output is emotion labels.

[1155] Step 7:

[1156] The server incorporates the emotion analysis results into the speech model. By incorporating emotion labels into the speech model, it becomes possible to generate speech that reflects the user's emotional state. The input data consists of the speech model and emotion labels, and the output is the speech model that reflects the emotions.

[1157] Step 8:

[1158] The server describes the metadata of the generated voice model and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model. The input is the metadata of the voice model, and the output is an NFT registered on the blockchain.

[1159] Step 9:

[1160] When a user uses the real-time speech conversion function, the device captures audio data in real time and streams it to the server. The input is real-time audio data, and the output is streamed audio data.

[1161] Step 10:

[1162] The server analyzes real-time audio data it receives and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, high-load processing is efficiently executed using GPU resources in the cloud. The input data consists of real-time audio data and a speech model, while the output is the converted audio data.

[1163] Step 11:

[1164] The server sends the converted audio data back to the user in real time. The terminal plays the received converted audio data through speakers or headphones. The input is the converted audio data, and the output is the played audio.

[1165] This ensures that specific data processing and calculations are performed at each processing step, ultimately resulting in a system that allows users to smoothly utilize voice conversion.

[1166] (Application Example 2)

[1167] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1168] In traditional online shopping, it was difficult to provide personalized voice models or emotionally resonant voices for virtual assistants when they interacted with users. This resulted in a mechanical user experience, leading to decreased emotional satisfaction and user engagement. Furthermore, there was a lack of efficient methods for handling the high-load processing required for real-time voice conversion.

[1169] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1170] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate individual voice models, means for registering the generated voice models as NFTs on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice models and sentiment analysis, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time and appropriately adjusting the voice based on the user's emotional state. This enables interactive and emotional responses by a personalized voice assistant in virtual stores, improving the user experience. Furthermore, it allows for the efficient distribution and execution of high-load cloud-based processing.

[1171] "Means for receiving voice samples from users" refers to devices or software that provide an interface for users to record their own voice samples and send them to the system.

[1172] "Means for analyzing received audio samples and generating individual audio models" refers to devices or algorithms that extract feature quantities from audio samples and generate user-specific audio models.

[1173] "Methods for registering generated voice models as NFTs on the blockchain" refers to devices or software that generate metadata for a voice model as an NFT and register it on the blockchain using smart contracts.

[1174] "Means for receiving real-time voice data and performing voice conversion by applying a generated voice model and sentiment analysis" refers to a device or software that receives real-time voice data from a user and converts it using a user-specific voice model and sentiment analysis engine.

[1175] "Means of performing speech conversion processing using cloud-based GPU resources" refers to devices or software that efficiently perform the high-load processing of real-time speech conversion by utilizing cloud-based high-performance GPU resources.

[1176] "Means for returning converted audio data to the user in real time and appropriately adjusting the audio based on the user's emotional state" refers to a device or software that immediately returns converted audio data to the user and further appropriately adjusts the audio based on the user's emotional state.

[1177] This invention realizes a system in which a virtual assistant with an individualized voice model guides users while they enjoy shopping in a virtual store. The overall system configuration and specific embodiments are described in detail below.

[1178] Server-side processing

[1179] User registration and receiving audio samples

[1180] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[1181] Voice model generation, registration, and sentiment analysis

[1182] The server analyzes the received audio sample and generates an individual audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion analysis engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific features and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[1183] Real-time speech conversion and emotion recognition

[1184] Real-time speech-to-text conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results to perform the conversion. During this process, GPU resources in the cloud (e.g., Google Cloud Speech-to-Text) are utilized to efficiently execute complex conversion processes. The converted audio data is then returned to the user in real time.

[1185] Terminal-side processing

[1186] Recording and sending audio samples

[1187] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[1188] Real-time speech conversion operation

[1189] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[1190] User-side operations

[1191] User registration and provision of audio samples

[1192] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[1193] Using real-time speech conversion

[1194] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[1195] Specific example

[1196] When a user records an audio sample and sends it to the server

[1197] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates an individual voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[1198] When a user uses real-time speech conversion

[1199] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[1200] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[1201] Examples of prompt statements

[1202] For example, if a user asks the virtual assistant, "Tell me more about this product," the following prompt will be generated.

[1203] The user asked, "Tell me more about this product."

[1204] Emotion analysis result: Very interesting

[1205] Generated audio: An audio that enthusiastically explains detailed information.

[1206] This prompt allows for the generation of appropriate messages in real time, based on sentiment analysis results, and provides them to the user.

[1207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1208] Step 1:

[1209] To register an account, users access the account creation screen through the application, enter the required information, and submit it.

[1210] Input: User information (name, email address, password, etc.)

[1211] Output: Unique User ID

[1212] Specific operation: The terminal sends the entered user information to the server, which stores this information in a database and generates a unique user ID.

[1213] Step 2:

[1214] The user records an audio sample. Once recording is complete, the device sends the audio sample data to the server.

[1215] Input: User voice sample

[1216] Output: Temporarily saved audio sample

[1217] Specific operation: The terminal records an audio sample, temporarily stores it, and then provides an interface for sending it to the server. The server receives the audio sample and temporarily stores it.

[1218] Step 3:

[1219] The server analyzes the received audio samples and generates individual audio models. During this process, it uses AI models to extract feature vectors from the audio samples.

[1220] Input: Audio sample

[1221] Output: Individual speech models

[1222] Specific operation: The server uses an AI library for speech generation to analyze speech samples, extract features, and then generate individual speech models.

[1223] Step 4:

[1224] The metadata of the generated voice model is registered on the blockchain as an NFT.

[1225] Input: Voice model

[1226] Output: NFT-registered voice models

[1227] Specific operation: The server generates voice model information as an NFT and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model.

[1228] Step 5:

[1229] To utilize the real-time voice conversion function, the user provides voice data via the microphone. The device then streams this real-time voice data to the server.

[1230] Input: Real-time audio data

[1231] Output: Streamed audio data

[1232] Specific operation: The device captures audio data in real time and streams it to the server.

[1233] Step 6:

[1234] The server analyzes the received real-time audio data and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, it utilizes GPU resources in the cloud to perform high-speed conversion.

[1235] Input: Real-time audio data, audio model, sentiment analysis results

[1236] Output: Converted audio data

[1237] Specific operation: The server uses cloud-based GPU resources to analyze and convert real-time audio data, and then applies the sentiment analysis results to adjust the audio.

[1238] Step 7:

[1239] The converted audio data is sent back to the user in real time, and the audio is appropriately adjusted based on the user's emotional state.

[1240] Input: Converted audio data, user's emotional state

[1241] Output: Real-time adjusted converted audio

[1242] Specific operation: The server immediately returns the converted audio data to the user and adjusts the audio appropriately according to the user's emotional state. The terminal receives this data and plays it back in real time through speakers or headphones.

[1243] In this way, by sequentially executing the specific processes performed at each step, the virtual assistant can provide a personalized voice model and an interactive user experience based on sentiment analysis.

[1244] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1245] The data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1246] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1247] [Fourth Embodiment]

[1248] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1249] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1250] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1251] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1252] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1253] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1254] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1255] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1256] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1257] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1258] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1259] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1260] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1261] The system of the present invention can create individual voice models using a voice generation AI, manage their rights using NFTs, and perform real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described below.

[1262] Server-side processing

[1263] User registration and receiving audio samples

[1264] The server first receives a user registration request. Once the user creates an account, the server stores this information in a database and generates a unique user ID. It then receives and temporarily stores a voice sample submitted by the user. The voice sample is used to understand the user's specific voice characteristics and speaking style.

[1265] Voice model generation and registration

[1266] The server analyzes the received audio sample and generates an audio model using AI technology. The generated audio model has user-specific characteristics and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[1267] Real-time speech conversion

[1268] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies a generated speech model to perform the conversion. During this process, GPU resources in the cloud are utilized to efficiently execute complex conversion operations. The converted audio data is then returned to the user in real time.

[1269] Terminal-side processing

[1270] Recording and sending audio samples

[1271] The user records an audio sample using the microphone on their smartphone or computer. Once the recording is complete, the device provides an interface for sending this data to the server.

[1272] Real-time speech conversion operation

[1273] When a user utilizes the real-time speech conversion function, the device streams audio data to the server in real time. After the server performs the conversion process, the device plays the converted audio in real time.

[1274] User-side operations

[1275] User registration and provision of audio samples

[1276] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[1277] Using real-time speech conversion

[1278] The user selects a conversion function and provides the audio in real time. The converted audio, processed on the server side, is played back on the device in real time and can be used directly by the user.

[1279] Specific example

[1280] When a user records an audio sample and sends it to the server

[1281] The user creates an account and records a 60-second audio sample following the application's instructions. Once recording is complete, the device sends the audio data to the server. The server analyzes the received audio sample and generates an audio model. The metadata of the audio model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[1282] When a user uses real-time speech conversion

[1283] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a generated speech model. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is sent back to the device in real time for the user to use.

[1284] This system provides comprehensive support for the entire process, from voice model generation to rights management and real-time voice conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[1285] The following describes the processing flow.

[1286] Server-side processing

[1287] Step 1:

[1288] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID.

[1289] Step 2:

[1290] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format.

[1291] Step 3:

[1292] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample, which are then input into the AI ​​model.

[1293] Step 4:

[1294] The server describes the generated voice model information as metadata and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[1295] Step 5:

[1296] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[1297] Step 6:

[1298] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion is performed by applying a speech model, generating audio data at high speed.

[1299] Step 7:

[1300] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[1301] Terminal-side processing

[1302] Step 1:

[1303] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[1304] Step 2:

[1305] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[1306] Step 3:

[1307] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[1308] Step 4:

[1309] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[1310] Step 5:

[1311] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[1312] User-side operations

[1313] Step 1:

[1314] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[1315] Step 2:

[1316] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[1317] Step 3:

[1318] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[1319] Step 4:

[1320] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[1321] (Example 1)

[1322] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1323] Existing voice conversion systems lack sufficient security and reliability in terms of the individuality and copyright management of voice data, and they have difficulty responding quickly to high-load voice conversion processing. Furthermore, there is a need for technology that can efficiently perform real-time voice conversion while protecting the copyright of voice data provided by users.

[1324] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1325] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model on a blockchain as a digital proof, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using remote computing resources, and means for returning the voice-converted data to the user in real time. This makes it possible to efficiently perform high-load voice conversion processing while ensuring the individuality of the voice data and copyright protection.

[1326] "Users" refer to individuals or companies that utilize the system, including those who provide voice samples or use real-time voice conversion.

[1327] A "voice sample" refers to data consisting of a user's own recorded voice, and is the basic data used to generate a voice model.

[1328] A "voice model" is a digital acoustic profile generated from analyzed voice samples, and refers to a model that possesses specific voice characteristics or speaking style traits.

[1329] "Digital proof means" refers to a technology that registers the metadata of a generated voice model on a distributed ledger technology (blockchain) to digitally prove ownership and usage rights.

[1330] "Blockchain" refers to a distributed ledger technology composed of multiple nodes, designed to ensure data transparency and prevent tampering.

[1331] "Real-time audio data" refers to audio data provided by users in real time, which is immediately converted into speech.

[1332] "Remote computing resources" refer to high-performance computing resources (e.g., GPUs) located in the cloud, which are used to efficiently distribute and execute high-load computational processes such as speech conversion.

[1333] "Speech conversion processing" refers to the process of applying a generated speech model to received real-time speech data and converting it into different speech characteristics.

[1334] The system of the present invention utilizes a speech generation AI to create individual speech models, manages their rights as digital proof-of-services (NFTs), and performs real-time speech conversion using cloud-based remote computing resources. This embodiment includes the processes of receiving speech samples, generating and registering speech models, and real-time speech conversion.

[1335] Server-side processing

[1336] The server first receives a voice sample from the user. The received voice sample is temporarily stored and analyzed in detail using AI technology. Deep learning frameworks such as TensorFlow and PyTorch are used for this analysis. As a result of the analysis, a user-specific voice model is generated. This voice model is described as metadata and registered on the blockchain as a means of digital proof. Distributed ledger technology is used as the blockchain platform. This digitally proves ownership and usage rights of the voice model.

[1337] Next, in the real-time speech conversion process, the user sends voice data to the server in real time via their device. The server analyzes the received voice data and performs speech conversion by applying a pre-generated voice model. During this process, a high-performance GPU in the cloud (such as NVIDIA Tesla V100) is used to efficiently perform the speech conversion. The converted voice data is then returned to the user in real time.

[1338] Terminal-side processing

[1339] The device provides an interface for recording user-provided audio samples and sending them to the server. When the user records an audio sample and presses the send button, this audio data is sent to the server. Furthermore, when the user utilizes the real-time voice conversion function, the device streams the audio data in real time via the microphone and plays the converted audio in real time.

[1340] User-side operations

[1341] Users create an account through the application and proceed to a page for providing a voice sample. By recording and submitting a voice sample according to the instructions, they register their voice with the system. They can also select the real-time voice conversion function and provide their voice in real time. The converted voice, processed on the server side, is played back on the device in real time and can be used directly by the user.

[1342] Specific example

[1343] 1. Prompt for user registration and voice sample provision

[1344] Launch the application and create an account. Then, follow the instructions to record a 60-second voice sample.

[1345] 2. Prompts for real-time speech conversion

[1346] To use the real-time speech conversion feature, prepare your microphone and press the conversion button in the application. Your speech will be converted in real time.

[1347] As described above, the system of the present invention provides a series of processes in which a user provides a voice sample, a voice model is generated and registered based on it, and voice conversion is performed in real time. This makes it possible to achieve efficient voice conversion while ensuring the security and reliability of voice data.

[1348] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1349] Step 1:

[1350] The user enters information such as their name, email address, and password on the account creation screen and presses the submit button. The entered information is sent from the device to the server. The server stores the received information in a database and generates a unique user ID. This is to manage the user's basic information, and the database includes the user ID, name, email address, etc. The user ID generated by the server is output.

[1351] Step 2:

[1352] The user proceeds to the audio sample provision page and records an audio sample using their device's microphone. Once recording is complete, the user presses the send button, and the audio data is sent from the device to the server. The server receives the audio data and temporarily stores it. This audio data is the basis for analyzing the user's voice quality and speaking style. An output confirms that the received audio data has been saved to the server.

[1353] Step 3:

[1354] The server analyzes the stored audio samples and generates an audio model using AI technology. Specifically, it extracts audio features using deep learning frameworks such as TensorFlow and PyTorch to generate an acoustic model. This model has user-specific audio characteristics. The generated audio model is output as a result of the analysis.

[1355] Step 4:

[1356] The server creates metadata for the generated voice model and registers it on the blockchain as a digital proof of value (NFT). A smart contract is used in this process, and the voice model's metadata is recorded on the blockchain platform (e.g., Ethereum or Hyperledger). This digitally proves ownership and usage rights of the voice model. The generated voice model's metadata and registration completion information are output.

[1357] Step 5:

[1358] If a user desires real-time speech conversion, they launch the application on their device and provide real-time audio data through the microphone. The device streams the real-time audio data to the server. The server receives this audio data and applies a pre-generated speech model to perform real-time speech conversion. This process is efficiently performed using a high-performance GPU in the cloud (e.g., NVIDIA Tesla V100). The converted audio data is output and immediately returned to the user.

[1359] Step 6:

[1360] Once the converted audio data is sent back to the device, the device plays this data in real time. The user can immediately listen to the converted audio. This process continues as long as the user continues to provide audio in real time. The output converted audio data is played back to the user in real time.

[1361] (Application Example 1)

[1362] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1363] This invention relates to a system that uses a speech generation AI to create individual speech models and converts speech in real time based on those models. Conventional technologies have problems with inefficient processing, requiring significant computational resources and time for creating speech models and real-time speech conversion. Furthermore, the rights management of speech models is ambiguous, creating risks of theft and misuse. In addition, it has been difficult for users to control virtual characters with their own voices. This invention aims to solve these problems and provide a highly efficient and secure speech conversion system.

[1364] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1365] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate a voice model, means for registering the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This ensures reliable rights management of the voice model and enables high-speed and highly efficient real-time voice conversion. Furthermore, the generated voice model enables a virtual character to perform in real time using the user's voice.

[1366] "Means for receiving audio samples from users" refers to means for recording audio data provided by users and sending it to the server.

[1367] "Means for analyzing received audio samples and generating audio models" refers to a method by which a server analyzes received audio data and constructs individual audio models based on that data.

[1368] "Method for registering a generated voice model as an NFT on the blockchain" refers to a method for recording the metadata of a generated voice model on the blockchain as an NFT and digitally proving the rights to it.

[1369] "Means for receiving real-time audio data and performing audio conversion by applying a generated audio model" refers to means for receiving audio data provided by the user in real time and performing conversion processing by applying an existing audio model.

[1370] "A means of performing speech conversion processing using GPU resources on the cloud" refers to a method of efficiently distributing the computational load of speech conversion processing by utilizing GPU resources located in a cloud environment.

[1371] "A means of sending the converted audio data back to the user in real time" refers to a means of immediately delivering the converted audio data to the user's device.

[1372] "A means of enabling a virtual character to perform in real time using a user's voice by utilizing a generated voice model" refers to a method of enabling a virtual character to play back the user's voice in real time using live streaming or recording, by leveraging a generated voice model.

[1373] This invention relates to a system that generates individual voice models using voice samples provided by the user and performs real-time voice conversion using these models. The overall system configuration and specific embodiments are described below.

[1374] Server-side processing

[1375] User registration and receiving audio samples

[1376] The server accepts users creating an account and recording and submitting a voice sample. The user's account information and voice sample are stored in the server's database, and a unique user ID is generated. This process identifies each individual user.

[1377] Voice model generation and NFT registration

[1378] The received audio samples are analyzed using AI technology to generate individual voice models. These voice models reproduce the user's specific voice characteristics and speaking style. Furthermore, the metadata of these generated voice models is registered on the blockchain as NFTs, digitally proving the copyright and usage rights of the voice models. Smart contract technology is used to ensure the secure management of these rights.

[1379] Real-time speech conversion

[1380] When a user utilizes the real-time speech conversion function, the server receives streaming audio data from the user and applies a generated speech model to perform the conversion. This process leverages GPU resources in the cloud to achieve high-speed and efficient speech conversion. The converted audio data is returned to the user in real time.

[1381] Terminal-side processing

[1382] Recording and sending audio samples

[1383] The terminal provides an interface for the user to record voice samples. The recorded voice samples are sent to a server and used for analysis and voice model generation.

[1384] Real-time speech conversion operation

[1385] When a user utilizes the real-time voice conversion function, the device acquires voice data via the microphone and streams it to the server. After the voice conversion process on the server is complete, the converted audio is played back in real time from the device.

[1386] User-side operations

[1387] User registration and provision of audio samples

[1388] Users create an account using the application, record a voice sample, and send it to the system. At this stage, a voice model is generated and registered on the blockchain as an NFT.

[1389] Using real-time speech conversion

[1390] When a user utilizes the real-time voice conversion function, the device sends audio to the server in real time and receives the converted audio in real time. This allows users to apply their own voice to virtual characters and perform with them.

[1391] Program Processing Description

[1392] This system uses Python, the requests library, and the sounddevice library to record, send, and perform real-time speech conversion of audio samples. On the server side, cloud-based GPU resources are used to process the speech conversion at high speed. The generated speech models are then registered on the blockchain as NFTs using smart contracts.

[1393] Specific example

[1394] For example, when a user uses a virtual performance application, the following process takes place:

[1395] 1. The user creates an account, records an audio sample according to the instructions, and sends it to the server.

[1396] 2. The server receives the voice sample, analyzes it, and generates a voice model.

[1397] 3. The voice model will be registered on the blockchain as an NFT.

[1398] 4. Users utilize a real-time voice conversion function to apply their own voice to a virtual character and perform.

[1399] Example of a prompt:

[1400] This program records a voice sample provided by the user, sends it to an API server to generate a voice model, and further includes the process of converting and playing back the voice in real time. Below is an example of the code that records the user's voice and sends the sample to the server.

[1401] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1402] Step 1:

[1403] User registration and recording and submission of voice samples.

[1404] The user accesses the application using their device and creates an account. Once the user enters and submits their account information (username, password, etc.), the server receives it and stores it in its database. Next, the user records a voice sample using their device's microphone and sends the data to the server. The server receives this voice sample and temporarily stores it for analysis.

[1405] Input: User account information and voice sample

[1406] Output: User ID and voice sample data

[1407] Step 2:

[1408] Voice model generation and NFT registration

[1409] The server analyzes the received audio sample and generates a voice model using an AI model based on that data. This voice model reproduces the user's specific voice quality and speaking style. The metadata of the generated voice model is registered on the blockchain as an NFT, where digital copyright management takes place.

[1410] Input: Audio sample data

[1411] Output: Voice model registered as an NFT

[1412] Step 3:

[1413] Preparing for real-time speech conversion

[1414] The user selects the real-time speech conversion function using their device. The device uses its microphone to acquire voice data in real time and prepares to send it to the server. At this time, the server prepares to use the already generated voice model.

[1415] Input: User's real-time voice data, existing voice model

[1416] Output: Real-time streaming start notification

[1417] Step 4:

[1418] Receiving and converting real-time audio data

[1419] The terminal sends real-time audio data from the user to the server in streaming format. The server receives this audio data and applies a generative AI model to perform real-time speech conversion. This process utilizes GPU resources in the cloud to achieve high-speed and efficient speech conversion.

[1420] Input: Real-time audio data sent by the user

[1421] Output: Converted audio data

[1422] Step 5:

[1423] Return and playback of the converted audio data

[1424] The server immediately returns the converted audio data to the user's terminal. The terminal plays this received data in real time, allowing the user to instantly check the converted audio. This process enables the virtual character to perform in real time using the user's voice.

[1425] Input: Converted audio data

[1426] Output: Audio played back in real time

[1427] Through these steps, individual voice models are generated based on voice samples provided by the user, and these models are used to achieve real-time voice conversion. Furthermore, the copyright of the generated voice models can be registered as NFTs on the blockchain, allowing for digital rights management.

[1428] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1429] The system of this invention creates individual voice models using a voice generation AI and an emotion engine, manages their rights using NFTs, and performs real-time voice conversion using cloud-based GPU resources. The overall system configuration and specific embodiments are described in detail below.

[1430] Server-side processing

[1431] User registration and receiving audio samples

[1432] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[1433] Voice model generation, registration, and sentiment analysis

[1434] The server analyzes the received audio sample and generates an audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific characteristics and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[1435] Real-time speech conversion and emotion recognition

[1436] Real-time speech conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results from the sentiment engine to perform the conversion. In this process, GPU resources in the cloud are utilized to efficiently execute the complex conversion process. The converted audio data is returned to the user in real time.

[1437] Terminal-side processing

[1438] Recording and sending audio samples

[1439] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[1440] Real-time speech conversion operation

[1441] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[1442] User-side operations

[1443] User registration and provision of audio samples

[1444] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[1445] Using real-time speech conversion

[1446] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[1447] Specific example

[1448] When a user records an audio sample and sends it to the server

[1449] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates a voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[1450] When a user uses real-time speech conversion

[1451] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[1452] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[1453] The following describes the processing flow.

[1454] Server-side processing

[1455] Step 1:

[1456] The server accepts user registration requests. When a user enters the required information on the account creation screen and presses the submit button, the server saves that information to its database and generates a unique user ID.

[1457] Step 2:

[1458] The server receives the audio sample sent by the user. The audio sample is temporarily stored in a file format. After saving, a confirmation message is sent to the user.

[1459] Step 3:

[1460] The server analyzes the received audio sample and generates an audio model using an AI model. This analysis process extracts feature elements from the audio sample.

[1461] Step 4:

[1462] The server uses an emotion engine to analyze the user's emotions from the voice sample. The results of the emotion analysis are reflected in the voice model.

[1463] Step 5:

[1464] The server describes the metadata of the generated voice model and executes a smart contract on the blockchain to generate the NFT. The metadata includes the voice model's features and owner information.

[1465] Step 6:

[1466] The server receives real-time speech conversion requests. When the user initiates real-time speech conversion, audio data is streamed from the device.

[1467] Step 7:

[1468] The server passes the received audio data in real time to GPU resources in the cloud for speech conversion processing. The conversion process is performed by applying the speech model and sentiment analysis results, generating audio data at high speed.

[1469] Step 8:

[1470] The server sends the converted audio data back to the terminal in real time. The conversion results are transmitted with low latency, allowing users to use the converted audio without any noticeable difference.

[1471] Terminal-side processing

[1472] Step 1:

[1473] The device provides the user with an account creation screen. Once the user enters the required information and presses the submit button, the device sends that information to the server.

[1474] Step 2:

[1475] The device accepts voice sample recordings from the user. When the user presses the record button, the voice sample is recorded via the microphone and temporarily saved.

[1476] Step 3:

[1477] The device sends the recorded audio sample to the server. After the transmission is complete, a confirmation message is displayed to the user.

[1478] Step 4:

[1479] The device provides real-time voice conversion functionality. When the user presses the conversion start button, it collects real-time voice data from the microphone and streams it to the server.

[1480] Step 5:

[1481] The terminal receives the converted audio data sent back from the server. The received data is played back in real time through the speaker or headphones, allowing the user to check the converted audio.

[1482] User-side operations

[1483] Step 1:

[1484] The user opens the application, enters the required information on the account creation screen, and submits it. This registers the user information on the server.

[1485] Step 2:

[1486] The user proceeds to the menu for recording an audio sample and presses the record button to record an audio sample. Once recording is complete, they press the send button to send the audio data to the server.

[1487] Step 3:

[1488] The user utilizes the real-time speech conversion function. When the conversion start button is pressed, the user begins speaking through the microphone, and the audio data is sent to the server in real time.

[1489] Step 4:

[1490] The user listens to the converted audio in real time. Once the conversion process is complete, the audio data sent back from the server is played through the device.

[1491] (Example 2)

[1492] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1493] In recent years, speech generation technology has advanced dramatically, but there is a need for a method to efficiently generate speech models that reflect the individual voice quality and emotions of each user, and to clearly manage their ownership and usage rights. Furthermore, real-time, high-quality speech conversion requires significant computing resources, necessitating systems that can process this efficiently.

[1494] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to extract features and generate a voice model, means for analyzing the user's emotions from the voice data and reflecting the results in the voice model, means for registering the metadata of the generated voice model as an NFT on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice model and emotion analysis results, means for executing high-load voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time. This enables the efficient generation of voice models that reflect the user's individual voice quality and emotions, and high-quality voice conversion in real time.

[1495] A "user" is a person or organization that uses the system and is the entity that performs operations such as providing voice samples or transmitting voice data in real time.

[1496] A "voice sample" is audio data that a user records and provides to the system, and is used to understand the user's voice quality and speaking style.

[1497] "Features" are acoustic characteristics extracted from audio data and are used to generate audio models. Examples include Mel-frequency cepstrum coefficients (MFCCs).

[1498] A "voice model" is a model generated by analyzing received voice samples and based on their features, representing a digital voice that reflects the user's voice quality and emotions.

[1499] "Emotion analysis" is the process of estimating a user's emotional state from voice data, and is performed using an emotion engine.

[1500] "NFT" stands for Non-Fungible Token, a technology used to prove ownership and usage rights of digital data on the blockchain.

[1501] Blockchain is a system for securely and decentralizedly managing digital data, and it is a technology that records data in a series of "blocks" and links them together in a chain.

[1502] "Real-time audio data" refers to audio data provided by the user in real time through a microphone, which is then sent to a server and analyzed and converted in real time.

[1503] "GPU resources" refer to the computing power of a graphics processing unit and are resources used to efficiently perform computationally intensive tasks.

[1504] "Voice conversion" is the process of analyzing received real-time voice data and converting it by applying the generated voice model and sentiment analysis results.

[1505] "Metadata" refers to supplementary materials containing information about the voice model, and is data registered on the blockchain as an NFT.

[1506] The system of the present invention combines speech generation technology and emotion analysis technology, registers them as NFTs, and performs speech conversion in real time. The overall system configuration and specific embodiments are described in detail below.

[1507] Server-side processing

[1508] The server performs the following main tasks:

[1509] 1. User registration and receipt of audio samples

[1510] The server receives account creation requests from users and stores the entered information in a database (e.g., MySQL or PostgreSQL). It generates a user ID, receives a voice sample, and temporarily stores it in cloud storage (e.g., Amazon S3 or Google Cloud Storage).

[1511] 2. Generation of voice models and sentiment analysis

[1512] The server uses a speech analysis library (e.g., Librosa or PyDub) to analyze the received audio sample and extract features. Based on the extracted features, it uses a generative AI model (e.g., TensorFlow or PyTorch) to generate an audio model. An emotion engine (e.g., EmotionAI) analyzes the emotions from the audio data and incorporates the results into the audio model.

[1513] 3. Register the voice model as an NFT.

[1514] The server describes the metadata of the generated voice model and registers it on a blockchain (e.g., Ethereum or Solana) using a smart contract. This digitally proves ownership and usage rights of the voice model.

[1515] 4. Real-time speech conversion and emotion recognition

[1516] The server receives real-time audio data from the user, analyzes the data, applies a generated audio model and sentiment analysis results, and performs speech conversion. This process utilizes GPU resources in the cloud (e.g., NVIDIA GPU Cloud). The converted audio data is returned to the user in real time.

[1517] Terminal-side processing

[1518] The terminal performs the following main processes:

[1519] 1. Record and send audio samples.

[1520] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, which is then sent to the server. The user records a voice sample, which is temporarily stored and sent to the server once the recording is complete.

[1521] 2. Operating Real-Time Speech Conversion

[1522] The device has a real-time voice conversion function for the user. When the user provides voice in real time, the voice data is streamed to the server. After the conversion process is performed on the server side, the converted voice data is played back on the device.

[1523] User-side operations

[1524] The user performs the following main operations:

[1525] 1. User registration and provision of audio samples

[1526] Users create an account through the application and proceed to a page for providing a voice sample. They register their voice by recording a voice sample and submitting it to the system.

[1527] 2. Use of real-time speech conversion

[1528] The user selects the real-time voice conversion function within the application and provides voice in real time to the microphone. The voice data is processed on the server side, the emotional state is recognized, and the appropriately adjusted voice is played back in real time from the device.

[1529] Specific example

[1530] An example of the specific processing procedure is shown below.

[1531] When a user records an audio sample and sends it to the server

[1532] 1. The user enters their name, email address, password, etc. on the account creation screen and presses the submit button.

[1533] 2. The server saves the received information to the database and generates a unique user ID.

[1534] 3. The user follows the application's instructions and records a 60-second audio sample via the microphone.

[1535] 4. After recording is complete, the device uploads the audio file to the server.

[1536] 5. The server analyzes the audio data and generates an audio model.

[1537] 6. The emotion engine analyzes the audio data and incorporates the emotional information into the audio model.

[1538] 7. The server describes the metadata of the voice model and registers it on the blockchain.

[1539] 8. A completion notification is sent to the user.

[1540] When a user uses real-time speech conversion

[1541] 1. The user launches the application and selects the real-time speech conversion function.

[1542] 2. The device captures the user's real-time voice through the microphone and streams it to the server.

[1543] 3. The server analyzes the received audio data and performs speech conversion by applying the audio model and sentiment analysis results.

[1544] 4. The converted audio data is efficiently processed using GPU resources in the cloud.

[1545] 5. The converted audio data is sent back to the device in real time and played back through the speaker or headphones.

[1546] Examples of input prompt messages

[1547] "Please register a new account. Enter the required information and record and submit a 60-second voice sample."

[1548] "Please begin real-time speech conversion. When you speak into the microphone, your voice will be automatically sent to the server."

[1549] This system utilizes generative AI models and an emotion engine to consistently support the entire process, from generating voice models to rights management, emotion analysis, and real-time speech conversion. Furthermore, it leverages cloud-based GPU resources to efficiently perform advanced processing.

[1550] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1551] Step 1:

[1552] The user enters the required information on the account creation screen and presses the submit button. This information includes name, email address, and password. The server receives this information and stores it in the database. Based on the information stored in the database, the server generates a unique user ID and stores this user ID in the database. The input data is the user's registration information, and the output is the user ID stored in the database.

[1553] Step 2:

[1554] The user proceeds to a screen to record a voice sample. The user presses the record button to record a voice sample through the microphone. Once recording is complete, the device temporarily saves the voice sample data. The input is the user's voice data, and the output is the temporarily saved voice data.

[1555] Step 3:

[1556] The device sends temporarily stored audio data to the server. At the same time, metadata such as the audio file format (e.g., WAV format) and sampling rate is also sent. The server saves the received audio sample to cloud storage. The data received by the server is the audio sample, and the output is the audio data stored in cloud storage.

[1557] Step 4:

[1558] The server analyzes the audio sample and extracts features. Specifically, it uses an audio analysis library (e.g., Librosa or PyDub) to extract acoustic features. The extracted features include MFCC (Mel-frequency cepstrum coefficients) and zero crossover ratio. The input data is an audio sample, and the output is the extracted features.

[1559] Step 5:

[1560] The server generates a speech model using a generative AI model based on the extracted features. The speech model is constructed using AI model libraries such as TensorFlow or PyTorch. The input data consists of features, and the output is the speech model.

[1561] Step 6:

[1562] The server analyzes the user's emotions from the audio data. An emotion engine (e.g., EmotionAI) is used to estimate the emotional state within the audio data. The analysis results are output as emotion labels such as joy, sadness, and anger. The input is audio data, and the output is emotion labels.

[1563] Step 7:

[1564] The server incorporates the emotion analysis results into the speech model. By incorporating emotion labels into the speech model, it becomes possible to generate speech that reflects the user's emotional state. The input data consists of the speech model and emotion labels, and the output is the speech model that reflects the emotions.

[1565] Step 8:

[1566] The server describes the metadata of the generated voice model and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model. The input is the metadata of the voice model, and the output is an NFT registered on the blockchain.

[1567] Step 9:

[1568] When a user uses the real-time speech conversion function, the device captures audio data in real time and streams it to the server. The input is real-time audio data, and the output is streamed audio data.

[1569] Step 10:

[1570] The server analyzes real-time audio data it receives and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, high-load processing is efficiently executed using GPU resources in the cloud. The input data consists of real-time audio data and a speech model, while the output is the converted audio data.

[1571] Step 11:

[1572] The server sends the converted audio data back to the user in real time. The terminal plays the received converted audio data through speakers or headphones. The input is the converted audio data, and the output is the played audio.

[1573] This ensures that specific data processing and calculations are performed at each processing step, ultimately resulting in a system that allows users to smoothly utilize voice conversion.

[1574] (Application Example 2)

[1575] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1576] In traditional online shopping, it was difficult to provide personalized voice models or emotionally resonant voices for virtual assistants when they interacted with users. This resulted in a mechanical user experience, leading to decreased emotional satisfaction and user engagement. Furthermore, there was a lack of efficient methods for handling the high-load processing required for real-time voice conversion.

[1577] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1578] In this invention, the server includes means for receiving voice samples from a user, means for analyzing the received voice samples to generate individual voice models, means for registering the generated voice models as NFTs on the blockchain, means for receiving real-time voice data and performing voice conversion by applying the generated voice models and sentiment analysis, means for executing voice conversion processing using GPU resources on the cloud, and means for returning the converted voice data to the user in real time and appropriately adjusting the voice based on the user's emotional state. This enables interactive and emotional responses by a personalized voice assistant in virtual stores, improving the user experience. Furthermore, it allows for the efficient distribution and execution of high-load cloud-based processing.

[1579] "Means for receiving voice samples from users" refers to devices or software that provide an interface for users to record their own voice samples and send them to the system.

[1580] "Means for analyzing received audio samples and generating individual audio models" refers to devices or algorithms that extract feature quantities from audio samples and generate user-specific audio models.

[1581] "Methods for registering generated voice models as NFTs on the blockchain" refers to devices or software that generate metadata for a voice model as an NFT and register it on the blockchain using smart contracts.

[1582] "Means for receiving real-time voice data and performing voice conversion by applying a generated voice model and sentiment analysis" refers to a device or software that receives real-time voice data from a user and converts it using a user-specific voice model and sentiment analysis engine.

[1583] "Means of performing speech conversion processing using cloud-based GPU resources" refers to devices or software that efficiently perform the high-load processing of real-time speech conversion by utilizing cloud-based high-performance GPU resources.

[1584] "Means for returning converted audio data to the user in real time and appropriately adjusting the audio based on the user's emotional state" refers to a device or software that immediately returns converted audio data to the user and further appropriately adjusts the audio based on the user's emotional state.

[1585] This invention realizes a system in which a virtual assistant with an individualized voice model guides users while they enjoy shopping in a virtual store. The overall system configuration and specific embodiments are described in detail below.

[1586] Server-side processing

[1587] User registration and receiving audio samples

[1588] The server first receives a user registration request. When the user enters the required information on the account creation screen and presses the submit button, the server saves that information to the database and generates a unique user ID. Subsequently, it receives a voice sample sent by the user and temporarily stores it. The voice sample is used to understand the user's specific voice quality and speaking style.

[1589] Voice model generation, registration, and sentiment analysis

[1590] The server analyzes the received audio sample and generates an individual audio model using an AI model. The analysis process extracts features from the audio sample. Furthermore, an emotion analysis engine analyzes the user's emotions from the audio data and reflects the results in the audio model. The generated audio model possesses user-specific features and emotional states, and is registered on the blockchain as an NFT for copyright management. The server describes the generated audio model information as metadata and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the audio model.

[1591] Real-time speech conversion and emotion recognition

[1592] Real-time speech-to-text conversion is performed by the user sending audio data to a server in real time using an application. The server analyzes the received real-time audio data and applies the generated speech model and sentiment analysis results to perform the conversion. During this process, GPU resources in the cloud (e.g., Google Cloud Speech-to-Text) are utilized to efficiently execute complex conversion processes. The converted audio data is then returned to the user in real time.

[1593] Terminal-side processing

[1594] Recording and sending audio samples

[1595] The device provides the user with an account creation screen. The user enters the required information and presses the submit button, at which point the device sends that information to the server. The user then records a voice sample, which is temporarily stored. Once recording is complete, the device provides an interface for sending this data to the server.

[1596] Real-time speech conversion operation

[1597] When a user utilizes the real-time voice conversion function, the device streams voice data to the server in real time. After processing on the server side, the converted voice data is adjusted according to the user's emotional state and sent back in real time. The device receives the converted voice data and plays it back in real time through speakers or headphones.

[1598] User-side operations

[1599] User registration and provision of audio samples

[1600] Users create an account through the application and proceed to a page for submitting a voice sample. By recording and submitting a voice sample, they register their voice with the system.

[1601] Using real-time speech conversion

[1602] The user selects a conversion function and provides the audio in real time. The audio data is processed on the server side, the emotional state is recognized, and the appropriately adjusted audio is played back in real time from the device. The user can use the converted audio immediately.

[1603] Specific example

[1604] When a user records an audio sample and sends it to the server

[1605] The user creates an account and records a 60-second voice sample following the application's instructions. Once recording is complete, the device sends the voice data to the server. The server analyzes the received voice sample and generates an individual voice model using AI and an emotion engine. The metadata of the voice model is then registered on the blockchain as an NFT, and a completion notification is sent to the user.

[1606] When a user uses real-time speech conversion

[1607] The user launches the application and provides real-time audio data via the microphone. The device streams this audio data to the server. The server receives the audio data and performs speech conversion by applying a speech model and sentiment analysis. This conversion is performed using GPU resources in the cloud and is processed quickly. The converted audio data is adjusted based on sentiment and sent back to the device in real time, allowing the user to use the converted voice without feeling any discomfort.

[1608] This system provides comprehensive support for the entire process, from voice model generation and rights management to sentiment analysis and real-time speech conversion, offering users convenience and security. Furthermore, by utilizing cloud-based resources, it can efficiently handle high-load processing.

[1609] Examples of prompt statements

[1610] For example, if a user asks the virtual assistant, "Tell me more about this product," the following prompt will be generated.

[1611] The user asked, "Tell me more about this product."

[1612] Emotion analysis result: Very interesting

[1613] Generated audio: An audio that enthusiastically explains detailed information.

[1614] This prompt allows for the generation of appropriate messages in real time, based on sentiment analysis results, and provides them to the user.

[1615] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1616] Step 1:

[1617] To register an account, users access the account creation screen through the application, enter the required information, and submit it.

[1618] Input: User information (name, email address, password, etc.)

[1619] Output: Unique User ID

[1620] Specific operation: The terminal sends the entered user information to the server, which stores this information in a database and generates a unique user ID.

[1621] Step 2:

[1622] The user records an audio sample. Once recording is complete, the device sends the audio sample data to the server.

[1623] Input: User voice sample

[1624] Output: Temporarily saved audio sample

[1625] Specific operation: The terminal records an audio sample, temporarily stores it, and then provides an interface for sending it to the server. The server receives the audio sample and temporarily stores it.

[1626] Step 3:

[1627] The server analyzes the received audio samples and generates individual audio models. During this process, it uses AI models to extract feature vectors from the audio samples.

[1628] Input: Audio sample

[1629] Output: Individual speech models

[1630] Specific operation: The server uses an AI library for speech generation to analyze speech samples, extract features, and then generate individual speech models.

[1631] Step 4:

[1632] The metadata of the generated voice model is registered on the blockchain as an NFT.

[1633] Input: Voice model

[1634] Output: NFT-registered voice models

[1635] Specific operation: The server generates voice model information as an NFT and registers it on the blockchain using a smart contract. This digitally proves ownership and usage rights of the voice model.

[1636] Step 5:

[1637] To utilize the real-time voice conversion function, the user provides voice data via the microphone. The device then streams this real-time voice data to the server.

[1638] Input: Real-time audio data

[1639] Output: Streamed audio data

[1640] Specific operation: The device captures audio data in real time and streams it to the server.

[1641] Step 6:

[1642] The server analyzes the received real-time audio data and performs speech conversion by applying the generated speech model and sentiment analysis results. During this process, it utilizes GPU resources in the cloud to perform high-speed conversion.

[1643] Input: Real-time audio data, audio model, sentiment analysis results

[1644] Output: Converted audio data

[1645] Specific operation: The server uses cloud-based GPU resources to analyze and convert real-time audio data, and then applies the sentiment analysis results to adjust the audio.

[1646] Step 7:

[1647] The converted audio data is sent back to the user in real time, and the audio is appropriately adjusted based on the user's emotional state.

[1648] Input: Converted audio data, user's emotional state

[1649] Output: Real-time adjusted converted audio

[1650] Specific operation: The server immediately returns the converted audio data to the user and adjusts the audio appropriately according to the user's emotional state. The terminal receives this data and plays it back in real time through speakers or headphones.

[1651] In this way, by sequentially executing the specific processes performed at each step, the virtual assistant can provide a personalized voice model and an interactive user experience based on sentiment analysis.

[1652] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1653] The data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of the data generation model 58 is ChatGPT (Internet Search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1654] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1655] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1656] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1657] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1658] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1659] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1660] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1661] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1662] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1663] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1664] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1665] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1666] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1667] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1668] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1669] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1670] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1671] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1672] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[1673] The following is further disclosed regarding the embodiments described above.

[1674] (Claim 1)

[1675] A means of receiving audio samples from users,

[1676] A means for analyzing received audio samples and generating an audio model,

[1677] A method for registering the generated voice model as an NFT on the blockchain,

[1678] A means for receiving real-time audio data and performing speech conversion by applying a generated audio model,

[1679] A means of performing speech conversion processing using GPU resources in the cloud,

[1680] A means of sending the voice-converted data back to the user in real time,

[1681] A system that includes this.

[1682] (Claim 2)

[1683] The system according to claim 1, which generates metadata of a user's voice model as an NFT and manages the rights on a blockchain.

[1684] (Claim 3)

[1685] The system according to claim 1, which distributes high-load processing by utilizing GPUs on the cloud in real-time speech conversion processing.

[1686] "Example 1"

[1687] (Claim 1)

[1688] A means of receiving audio samples from users,

[1689] A means for analyzing received audio samples and generating an audio model,

[1690] A means of registering the generated voice model on the blockchain as a digital proof,

[1691] A means for receiving real-time audio data and performing speech conversion by applying a generated audio model,

[1692] A means of performing speech conversion processing using remote computing resources,

[1693] A means of sending the voice-converted data back to the user in real time,

[1694] A system that includes this.

[1695] (Claim 2)

[1696] The system according to claim 1, which generates metadata of a user's voice model as a digital proof and manages rights on a distributed ledger.

[1697] (Claim 3)

[1698] The system according to claim 1, which distributes high-load processing by utilizing remote computing resources in real-time speech conversion processing.

[1699] "Application Example 1"

[1700] (Claim 1)

[1701] A means of receiving audio samples from users,

[1702] A means for analyzing received audio samples and generating an audio model,

[1703] A method for registering the generated voice model as an NFT on the blockchain,

[1704] A means for receiving real-time audio data and performing speech conversion by applying a generated audio model,

[1705] A means of performing speech conversion processing using GPU resources in the cloud,

[1706] A means of sending the voice-converted data back to the user in real time,

[1707] A method for a virtual character to perform in real time using the user's voice, utilizing a generated voice model.

[1708] A system that includes this.

[1709] (Claim 2)

[1710] The system according to claim 1, which generates metadata of a user's voice model as an NFT and manages the rights on a blockchain.

[1711] (Claim 3)

[1712] The system according to claim 1, which distributes high-load processing by utilizing GPUs on the cloud in real-time speech conversion processing.

[1713] "Example 2 of combining an emotion engine"

[1714] (Claim 1)

[1715] A means of receiving audio samples from users,

[1716] A means for analyzing received audio samples, extracting features, and generating an audio model,

[1717] A method for analyzing user emotions from voice data and reflecting the results in a voice model,

[1718] A method for registering the metadata of the generated voice model as an NFT on the blockchain,

[1719] A means for receiving real-time voice data and performing voice conversion by applying the generated voice model and sentiment analysis results,

[1720] A means of performing high-load speech conversion processing using GPU resources in the cloud,

[1721] A means of returning the converted audio data to the user in real time,

[1722] A system that includes this.

[1723] (Claim 2)

[1724] The system according t...

Claims

1. A means of receiving audio samples from users, A means for analyzing received audio samples and generating an audio model, A method for registering the generated voice model as an NFT on the blockchain, A means for receiving real-time audio data and performing speech conversion by applying a generated audio model, A means of performing speech conversion processing using GPU resources in the cloud, A means of sending the voice-converted data back to the user in real time, A system that includes this.

2. The system according to claim 1, which generates metadata of a user's voice model as an NFT and manages the rights on a blockchain.

3. The system according to claim 1, which distributes high-load processing by utilizing GPUs on the cloud in real-time speech conversion processing.

Images