System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording a user's voice; means for transmitting the recorded voice data to a server; means for analyzing the voice data received by the server and converting the voice data into text data; means for generating voice data from the text data; means for storing the voice data with an identifier in a database; means for listing the voice data with an identifier in a marketplace; means for managing a transaction of the voice data in the marketplace; means for downloading the voice data purchased by the user to a terminal; and means for reproducing the voice data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional voice generation technologies face challenges when users suddenly lose the ability to speak or are absent, making it difficult to provide a substitute for the user. Furthermore, there is a high risk of the generated voice data being forged or misused. Furthermore, there is a lack of a platform that ensures appropriate security and reliability for selling users' voices as content. Therefore, there is a need for a system that can properly manage users' voice data and provide diverse ways to utilize it. [Means for solving the problem]

[0005] This invention provides a means for recording a user's voice and transmitting the recorded voice data to a server, and a means for the server to analyze the received voice data and convert it into text data. The server is equipped with a means for generating voice data from text data and assigning an identifier to the voice data. The voice data with the identifier is stored in a database, and through a means for listing the voice data on a marketplace, transactions of the voice data are managed and a means for downloading and playing the voice data is provided to purchasers. This allows users to safely manage their own voice data and play it in real time when needed. Furthermore, it becomes easier to set prices for voice data and manage transactions on the marketplace, providing new value for voice data.

[0006] An "audio recording means" is a device or software for recording a user's voice in digital form.

[0007] The "means for transmitting voice data" refers to a communication device or protocol for transmitting recorded voice data to a server via the Internet or the like.

[0008] A "server" is a computer system for analyzing, storing, playing back, and managing audio data.

[0009] The "means for analyzing voice data" refers to acoustic processing technology and software for converting the user's recorded voice data into text data.

[0010] The "means for converting into text data" is a technology that analyzes voice data and converts it into text-format data based on linguistic elements.

[0011] "Means for generating voice data" refers to voice synthesis technology and software for creating new voice data from text data.

[0012] "Means of assigning an identifier" refers to a technology that attaches personalized information such as an NFT to audio data to prove that the data is unique.

[0013] "Means of storing in a database" refers to a system that systematically stores and manages voice data with identifiers.

[0014] "Marketplace" means an online platform for buying and selling identified audio data.

[0015] "Means for Managing Transactions" means the software and protocols that govern the buying, selling, pricing, and fulfillment of transactions for audio data in the Marketplace.

[0016] "Means for downloading audio data" refers to a technology for transferring data to save audio data on the user's terminal.

[0017] "Means for reproducing audio data" refers to technology and devices that output stored audio data as audio through a playback device. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data. The system of the present invention solves the problem by having the user, server, and terminal work together as shown below.

[0040] User processing

[0041] Audio Recording and Transmission

[0042] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[0043] Buying and selling

[0044] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[0045] Server-side processing

[0046] Speech Analysis and Generation

[0047] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[0048] Assigning identifiers

[0049] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[0050] Marketplace operations

[0051] The server publishes the audio data that users have listed on the marketplace. When a user purchases the data, the audio data is downloaded to the buyer's device. The server also manages transactions, notifies the buyer, and processes the seller's compensation.

[0052] Terminal side processing

[0053] Download and play audio

[0054] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[0055] Data Management and Backup

[0056] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[0057] Specific examples

[0058] As an example, consider a scenario where a user wants to record a narration for a presentation. The user uses a dedicated application to record the audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. Users can access a marketplace where they can sell their own audio data or purchase audio data from other users. The audio data is downloaded and played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[0059] By implementing these functions, the present invention solves the problems associated with users' voices and provides new value to voice data.

[0060] The processing flow will be explained below.

[0061] User processing steps

[0062] Step 1:

[0063] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[0064] Step 2:

[0065] The user presses the "Start Recording" button to start recording the audio, and when they have finished speaking, they press the "Stop Recording" button to stop recording.

[0066] Step 3:

[0067] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[0068] Step 4:

[0069] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[0070] Server-side processing steps

[0071] Step 1:

[0072] The server receives the voice data sent by the user and temporarily stores it.

[0073] Step 2:

[0074] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[0075] Step 3:

[0076] The server generates new voice data based on the text data using generative AI technology.

[0077] Step 4:

[0078] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[0079] Step 5:

[0080] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[0081] Step 6:

[0082] The server publishes the audio data that users have put up for sale on the marketplace and manages transactions of the audio data. When a transaction is completed, the audio data is provided to the buyer and a reward is paid to the seller.

[0083] Terminal processing steps

[0084] Step 1:

[0085] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[0086] Step 2:

[0087] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[0088] Step 3:

[0089] When a user inputs text data in real time, the terminal transmits this data to the server and instantly plays back the generated voice data.

[0090] Step 4:

[0091] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[0092] In this way, a system is realized in which the user, server, and terminal work in cooperation to effectively manage and utilize the user's voice data.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] Counterfeiting and unauthorized use of voice data have become a problem in the current voice data trading market. Conventional technologies lack a method to guarantee the uniqueness and authenticity of data when users record and trade their own voice data. This reduces the reliability of voice data and poses a challenge to ensuring transparency in transactions. Furthermore, the user experience is insufficient in the generation and management of voice data itself, as well as in post-purchase usage.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes means for recording the user's voice, means for transmitting the recorded voice data to the data server, means for analyzing the voice data received by the data server and converting it into text data, means for generating voice data from the text data, means for assigning an identifier to the generated voice data, means for storing the identified voice data in a database, means for listing the identified voice data on an online market, means for managing transactions of voice data on the online market, means for downloading voice data purchased by the user to an end-user terminal, and means for playing the voice data. This ensures the uniqueness and authenticity of the voice data and enables transparent transactions. Furthermore, users can efficiently generate, manage, trade, and use voice data.

[0098] "User" refers to any person or entity that uses the System to record voice and make transactions or purchases.

[0099] "Audio data" refers to data that represents audio information recorded by a user in digital form.

[0100] "Data Server" refers to a computer system that receives, analyzes, and stores voice data sent by users and manages transactions of voice data.

[0101] "Analysis" refers to the process of converting received voice data into text data.

[0102] "Text data" refers to character information obtained by analyzing voice data.

[0103] "Speech generation" refers to the process of generating new voice data based on the text data obtained through analysis.

[0104] "Identifier" refers to a digital tag or code that is attached to generated audio data to ensure the uniqueness and authenticity of the audio data.

[0105] "Database" refers to a storage device or storage system for storing identified audio data.

[0106] "Electronic Marketplace" refers to an online platform for trading identified voice data.

[0107] "End-user device" refers to a user's device (e.g., smartphone, personal computer) used to record, transmit, download, or play audio data.

[0108] "Playback" refers to the process of outputting audio data downloaded to an end-user terminal as an acoustic signal.

[0109] MODE FOR CARRYING OUT THE INVENTION

[0110] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data.

[0111] System Overview

[0112] The system uses the following hardware and software:

[0113] The device used by the user (e.g. smartphone, PC)

[0114] Dedicated application for recording and playback

[0115] Data server (receives, analyzes, and generates voice data)

[0116] Speech recognition technology for speech analysis (e.g., Google Cloud Speech-to-Text API)

[0117] AI models for speech generation (e.g., OpenAI GPT-3)

[0118] Database (storing voice data and identifiers)

[0119] Electronic market (voice data trading platform)

[0120] User processing

[0121] The user first launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, and the user presses the "Start Recording" button to begin recording. When recording is finished, the user presses the "Stop Recording" button, and then the "Send" button to upload the audio data to the server. At this time, the data is automatically sent via the Internet.

[0122] Server-side processing

[0123] The server temporarily stores the received voice data and performs voice analysis. The Google Cloud Speech-to-Text API is used to convert the voice data into text. The text data is then input into an AI model such as OpenAI GPT-3 to generate new voice data. This generated voice data is given an identifier such as an NFT to prevent counterfeiting or fraudulent use, and is stored in a database.

[0124] Terminal side processing

[0125] When a user purchases audio data from an online marketplace, the data is downloaded to the user's device via the Internet. The downloaded audio data is played using the player function within a dedicated application. Real-time audio generation is also possible, and audio data is instantly generated and played based on text data entered by the user.

[0126] Specific examples

[0127] As an example, consider a scenario in which a user wants to record narration for a presentation. The user launches a dedicated application and presses the "Start Recording" button to record the narration. When the recording is complete, the user presses the "Stop Recording" and "Send" buttons to send the audio data to the server. The server analyzes the audio data, converts it into text data, and then generates new audio data. This audio data is assigned an identifier and stored in a database. The user can then list the audio data on an electronic marketplace.

[0128] Prompt Sentence Examples

[0129] Please provide detailed instructions on how users can use the dedicated application to record audio and send the audio data to the server. Please also include details of the interface and the names of the operation buttons.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] The user launches a dedicated application. The application screen displays a "Start Recording" button and a "Stop Recording" button. When the user presses the "Start Recording" button, the device's microphone is enabled and audio recording begins. The input is the user's voice, and the output is the recorded audio data.

[0133] Step 2:

[0134] When the user presses the "Stop Recording" button to end recording, the audio recording stops. The device temporarily stores this recording data in local storage. The input is an instruction to stop audio recording, and the output is the temporarily stored audio data.

[0135] Step 3:

[0136] The user presses the "Send" button to send the recorded data to the server. The device then uploads the audio data to the server via the Internet. The input is the temporarily saved audio data, and the output is the audio data sent to the server.

[0137] Step 4:

[0138] The server stores the received audio data in a temporary storage area. At this point, it performs a data integrity check to ensure that it is not corrupted. The input is the transmitted audio data, and the output is the audio data stored in the temporary storage area.

[0139] Step 5:

[0140] The server starts the speech analysis. Specifically, it converts the speech data into text data using the Google Cloud Speech-to-Text API. The input is the speech data in the temporary storage area, and the output is the analyzed text data.

[0141] Step 6:

[0142] The server inputs the text data into an AI speech generation model (e.g., OpenAI GPT-3) to generate new speech data. The input is the analyzed text data, and the output is the generated speech data.

[0143] Step 7:

[0144] The server assigns an identifier such as an NFT to the generated audio data. This procedure guarantees the uniqueness and authenticity of the audio data. The input is the generated audio data, and the output is audio data with an identifier.

[0145] Step 8:

[0146] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database.

[0147] Step 9:

[0148] Users access the marketplace to sell their own voice data or purchase other users' voice data. The marketplace provides an interface for managing transactions and setting prices. The input is a marketplace access instruction, and the output is transaction information.

[0149] Step 10:

[0150] When a user purchases audio data, the server downloads the audio data to the end-user terminal. The input is a purchase instruction, and the output is the audio data downloaded to the end-user terminal.

[0151] Step 11:

[0152] The device plays the downloaded audio data within a dedicated application. Pressing the play button starts audio playback. The input is the downloaded audio data, and the output is the played audio.

[0153] (Application example 1)

[0154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] Conventional voice authentication systems have had problems with the accuracy and security of voice data authentication, and are not resistant to fraudulent use or attacks. Furthermore, there is a lack of transparency in the management and trading of voice authentication data, making it difficult to ensure trust between users. Furthermore, in addition to voice authentication, marketplace functions to increase the added value of voice data are lacking.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0157] In this invention, the server includes a means for recording the user's voice, a means for transmitting the recorded voice data to the server, and a means for analyzing the received voice data and converting it into text data. This improves the accuracy of voice data authentication, prevents fraudulent use, and enables safe and efficient trading of voice data through the marketplace.

[0158] A "user" is an entity that uses the system and records, transmits, and plays back audio data.

[0159] "Voice data" is digital data that is a recording of a user's speech.

[0160] A "server" is a computer system that receives, analyzes, generates, identifies, stores, and manages voice data via a network.

[0161] "Analysis" is the process of processing received voice data and converting it into text data.

[0162] "Text data" is character information converted from audio data through analysis.

[0163] An "identifier" is unique information that is assigned to generated voice data to prevent counterfeiting or unauthorized use of the voice data.

[0164] "Marketplace" means an online platform where audio data is traded.

[0165] "Authentication" is the process of verifying a user's authenticity using recorded voice data.

[0166] "Matching" is the process of comparing generated and recorded audio data to confirm a match.

[0167] "Downloading" is the act of transferring audio data from a server to a user's terminal.

[0168] "Playback" refers to the act of outputting downloaded audio data as sound again on the terminal.

[0169] A system for implementing this invention requires the following components to record, transmit, analyze, generate, identify, store, and transact user voice data:

[0170] 1. User-side processing

[0171] Hardware and software:

[0172] Smartphone: Microphone, Storage

[0173] Dedicated applications: voice recording, transmission, player, etc.

[0174] First, users launch the dedicated application installed on their smartphone and tap the recording start button to record their own voice. When recording is finished, they press the send button to send the recorded voice data to the server. The voice data is automatically uploaded to the server via the Internet. Users can also access the marketplace to list their own voice data or purchase other voice data.

[0175] 2. Server-side processing

[0176] Hardware and software:

[0177] Cloud server: Database, speech analysis system, speech generation system, identifier assignment system, etc.

[0178] Google Cloud Speech-to-Text API: An analysis system for converting voice data into text data.

[0179] AI model: A system for generating new voice data from text data

[0180] The server receives the voice data sent by the user and temporarily stores it. The voice analysis system then converts the received voice data into text data. The text data is then input back into the voice generation system, and new voice data is generated using AI technology. A unique identifier is assigned to the generated voice data using an identifier assignment system. The voice data with the identifier assigned is stored in a database and made public on the marketplace. The marketplace manages downloads and transactions of voice data purchased by users.

[0181] 3. Processing on the user terminal side

[0182] Hardware and software:

[0183] Smartphone: Storage, player functions

[0184] Cloud service: Regular backup function

[0185] When a user purchases audio data from the marketplace, the audio data is downloaded to their smartphone. The downloaded audio data is played using the player function within the application. The audio data is also saved in local storage as needed and periodically backed up to the cloud.

[0186] Specific examples

[0187] For example, when a user wants to unlock their smartphone, they record their voice using a dedicated application, and after recording is complete, send it to the server. The server analyzes the voice and compares it with the newly generated voice data for authentication. At this time, the voice data with an identifier is stored in a database, preventing unauthorized use. If authentication is successful, the smartphone is unlocked.

[0188] Prompt Sentence Examples

[0189] Describe a scenario in which a user attempts to unlock a smartphone using a specific voice. Explain in detail the process from recording the user's voice and sending it to the server, to analyzing the voice on the server side, assigning an identifier, and finally multi-factor authentication.

[0190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0191] Step 1:

[0192] The user records audio

[0193] The user launches a dedicated application on their smartphone and taps the "Start Recording" button to record their own voice. When the recording is complete, they press the "Stop Recording" button. In this step, the voice input from the user's microphone is saved as digital voice data. Specifically, the input data is the user's voice, and the output is an audio file saved on the smartphone.

[0194] Step 2:

[0195] Sending audio data from the device to the server

[0196] The user taps the "Send" button to send the recorded audio data to the server. The smartphone application uploads the audio data to the server as an HTTP POST request. The input is the recorded audio data, and the output is the audio data saved on the server. Specifically, the application saves the audio file as a temporary file and sends it to the server via the Internet.

[0197] Step 3:

[0198] The server analyzes the voice data and converts it into text data.

[0199] The server analyzes the received voice data and converts it into text data using a voice analysis system (e.g., Google Cloud Speech-to-Text API). The input is voice data, and the output is the converted text data. Specifically, the server sends the voice data to the API and receives the converted text.

[0200] Step 4:

[0201] The server generates new voice data from the text data.

[0202] The server inputs the text data into a speech generation system to generate new speech data. A generative AI model is used here. The input is text data, and the output is newly generated speech data. Specifically, the AI ​​speech generation model analyzes the text data and generates a speech file.

[0203] Step 5:

[0204] The server assigns an identifier to the generated audio data.

[0205] The server uses an identification system to assign a unique identifier (e.g., NFT) to the generated audio data. The input is the generated audio data, and the output is audio data with an identifier. Specifically, the audio data is processed using a hash function or similar to generate the identifier.

[0206] Step 6:

[0207] Save the identified audio data in a database

[0208] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database. Specifically, the voice data is stored in a fixed format using a database management system.

[0209] Step 7:

[0210] Publishing audio data with identifiers to the marketplace

[0211] The server publishes the identified voice data to the marketplace, allowing users to purchase and trade. The input is the voice data stored in the database, and the output is the voice data posted on the marketplace. Specifically, the web server displays the voice data through the marketplace interface.

[0212] Step 8:

[0213] The user purchases the audio data and downloads it to their device.

[0214] Users purchase audio data from the marketplace and download it to their smartphone. The input is the audio data on the marketplace, and the output is the audio data stored on the smartphone. Specifically, the application downloads the audio data using an HTTP request and saves it in local storage.

[0215] Step 9:

[0216] Play audio data on the device

[0217] The user plays the downloaded audio data using the player function of the dedicated application. The input is the audio data stored in local storage, and the output is the played audio. Specifically, the application's audio player reads the audio data and sends it to the audio output device (speakers or earphones) for playback.

[0218] This enables users to perform secure voice authentication for multi-stage authentication, and enables safe and efficient transactions of voice data with identifiers.

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] The present invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and then stores and trades it in a database, and further combines it with an emotion engine to increase the naturalness and diversity of the voice data. The system of the present invention solves these problems by having the user, server, and terminal work together.

[0221] User processing

[0222] Audio Recording and Transmission

[0223] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[0224] Buying and selling

[0225] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[0226] Server-side processing

[0227] Speech Analysis and Generation

[0228] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[0229] Combining Emotion Engines

[0230] In this invention, an emotion engine is built into the server, which can recognize emotions from the user's voice data. The emotion engine acquires the user's emotion data and reflects it in the generated voice data. For example, if the user is speaking with a cheerful tone, that emotion can be reflected in the tone and pitch of the voice data.

[0231] Assigning identifiers

[0232] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[0233] Marketplace operations

[0234] The server publishes voice data that users have listed on the marketplace. When a user purchases a voice data item, the voice data is downloaded to the buyer's device. The server also manages transactions, notifies buyers, and processes sellers' compensation. It can also automatically categorize voice data using emotion data recognized by the emotion engine.

[0235] Terminal side processing

[0236] Download and play audio

[0237] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[0238] Data Management and Backup

[0239] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[0240] Specific examples

[0241] As an example, consider a scenario in which a user is recording narration for a presentation. The user uses a dedicated application to record audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. An emotion engine is also used to recognize the user's emotions and reflect them in the generated audio data. This data is then put up for sale on a marketplace where other users can purchase it. The audio data downloaded by the purchaser is then played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[0242] By implementing these functions, the present invention solves the problems of users' voices and provides new value to voice data. By combining it with an emotion engine, it becomes possible to generate even more natural and emotionally rich voice data.

[0243] The processing flow will be explained below.

[0244] User processing steps

[0245] Step 1:

[0246] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[0247] Step 2:

[0248] The user presses the "Start Recording" button to begin recording the audio.

[0249] Step 3:

[0250] When the user has finished speaking, he or she presses the "End Recording" button to stop recording.

[0251] Step 4:

[0252] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[0253] Step 5:

[0254] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[0255] ---

[0256] Server-side processing steps

[0257] Step 1:

[0258] The server receives the voice data sent by the user and temporarily stores it.

[0259] Step 2:

[0260] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[0261] Step 3:

[0262] The server uses generative AI technology to generate new voice data based on the text data, and is equipped with an emotion engine that recognizes the user's emotions from the voice data.

[0263] Step 4:

[0264] The emotion engine analyzes the recognized emotion data and reflects it in the generated voice data. For example, if the user is speaking with a happy expression, that emotion will be reflected in the tone and pitch of the voice data.

[0265] Step 5:

[0266] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[0267] Step 6:

[0268] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[0269] Step 7:

[0270] The server publishes the voice data that users have put up for sale on the marketplace and manages the transactions of the voice data. When a transaction is completed, the server provides the voice data to the buyer and pays a fee to the seller. The server also automatically categorizes the voice data using emotion data recognized by the emotion engine.

[0271] ---

[0272] Terminal processing steps

[0273] Step 1:

[0274] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[0275] Step 2:

[0276] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[0277] Step 3:

[0278] When a user inputs text data in real time, the device transmits the data to the server and instantly plays back the generated voice data, with the emotion engine reflecting appropriate emotions in the voice data based on the text data entered by the user.

[0279] Step 4:

[0280] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[0281] In this way, a system is realized in which the user, server, and terminal work together to effectively manage and utilize the user's voice data. In addition, the introduction of an emotion engine greatly improves the naturalness and emotional impact of the voice data.

[0282] Example 2

[0283] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0284] The present invention aims to generate more natural and diverse voice data by reflecting emotions when analyzing and generating voice data. Another objective of the present invention is to improve the reliability and convenience of voice data by realizing secure transaction management of voice data with assigned identifiers and cloud backup of voice data.

[0285] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing emotions from user voice data and reflecting the emotions in generated voice data, a means for saving multiple voice data input by the user in local storage and backing them up to the cloud as needed, and a means for managing transactions of voice data in the marketplace. This makes it possible to generate more natural and emotional voice data that reflects emotions, and also improve the reliability and safety of the voice data.

[0286] "User" refers to any individual or entity that records, transmits, buys, or sells audio data.

[0287] "Server" refers to a central management system for analyzing, generating, and managing voice data sent by users.

[0288] "Terminal" refers to the device used by a user (e.g., smartphone, tablet, computer, etc.).

[0289] "Audio Data" refers to digital files that contain recordings of a user's voice.

[0290] "Text data" refers to character information that has been converted from analyzed voice data.

[0291] "Identifier" refers to a unique identification number or NFT assigned to audio data.

[0292] "Marketplace" refers to an online platform for trading voice data.

[0293] An "emotion engine" refers to a system that recognizes a user's emotions from voice data and reflects them in the generated voice data.

[0294] "Local storage" refers to a storage area for storing data within a terminal.

[0295] "Cloud" refers to data storage provided on remote servers accessible over the Internet.

[0296] A "generative AI model" refers to an artificial intelligence algorithm for generating new voice data based on text data.

[0297] This invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and stores and trades it in a database. The main purpose is to increase the naturalness and diversity of voice data by combining it with an emotion engine. The system operates in cooperation with the user, server, and terminal.

[0298] User processing

[0299] Audio Recording and Transmission

[0300] The user starts the dedicated application and starts recording audio. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. This transmission process is carried out automatically via the Internet.

[0301] Buying and selling

[0302] Users can access the marketplace and list their own audio data or purchase audio data listed by other users. The marketplace provides an interface for setting the selling price. For example, a user can record a narration for a presentation and list it for 500 yen. The following is an example of a prompt:

[0303] "Hello everyone. Today I'd like to talk to you about our new project."

[0304] Server-side processing

[0305] Speech Analysis and Generation

[0306] The server receives the voice data sent by the user and temporarily stores it. It then uses Google Cloud's speech recognition API to analyze the voice data and convert it into text data. This text data is then input back into a speech generation system (e.g., a generative AI model) to generate new voice data. A generative AI model such as OpenAI's GPT-3 could be used.

[0307] Combining Emotion Engines

[0308] The server is equipped with an emotion engine that recognizes emotions from the user's voice data. It acquires emotion data using tools such as IBM Watson Tone Analyzer and reflects that emotion in the generated voice data. For example, if the user sounds like they're having fun, that emotion can be reflected in the tone and pitch of the voice data.

[0309] Assigning identifiers

[0310] The generated voice data is assigned an identifier, such as an NFT (Non-Fungible Token). The identifiers are managed using the Ethereum blockchain or similar to prove that the voice data is unique. After the identifier is assigned, the voice data is stored in a database.

[0311] Marketplace operations

[0312] The server publishes the voice data that users have put up for sale on the marketplace. When a user purchases voice data, the server downloads the data to the buyer's device, manages the transaction, notifies the buyer, and processes compensation for the seller. It also automatically sets the category of the voice data using data from the emotion engine.

[0313] Terminal side processing

[0314] Download and play audio

[0315] When a user purchases audio data from the marketplace, it is automatically downloaded to the device and can be played using the player function within the application. It is also possible to generate audio based on text data entered in real time and play it instantly.

[0316] Data Management and Backup

[0317] The user's device will save downloaded audio data to local storage as needed. Important audio data will also be backed up to the cloud periodically. Backups can be automated using the Google Drive API, for example.

[0318] Specific examples

[0319] For example, consider a scenario where a user is recording narration for a presentation. The user uses a dedicated app to record audio and send it to the server. The server then uses Google Cloud's speech recognition API to convert the audio data into text and generates new audio data using the OpenAI GPT-3 model. This audio data is assigned an NFT identifier using the Ethereum blockchain. An emotion engine is also used to recognize emotions and reflect them in tone and pitch. This data is then put up for sale on a marketplace where other users can purchase it. Buyers can play the downloaded audio data using the player function. Real-time audio generation is also possible, allowing for audio substitution even if the user is unable to speak.

[0320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0321] Step 1:

[0322] The user launches a dedicated application. They press the "Start Recording" button in the application and then speak into the microphone to record audio data. When recording is complete, they press the "Stop Recording" button. The input is the user's voice, and the output is an audio file (e.g., WAV format). Specifically, the user taps a button on the app screen on their smartphone.

[0323] Step 2:

[0324] The user presses the "Send" button to send the recorded audio data to the server. The data is automatically uploaded to the server via the Internet. The input is the recorded audio file, and the output is the audio file saved on the server. During the sending process, a progress bar is displayed on the screen.

[0325] Step 3:

[0326] The server temporarily stores the received voice data. The stored voice file is used as input data for analysis processing. Specifically, the received data is stored in a temporary folder.

[0327] Step 4:

[0328] The server converts the voice data into text data using Google Cloud's speech recognition API. Here, the voice file is input and the converted text data is output. A specific API call is made to convert the voice data into text information.

[0329] Step 5:

[0330] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate new voice data from text data. The input is text data, and the output is a new voice file. In this case, the voice data is generated by giving a prompt to the AI ​​model. For example, the following prompt sentence is used:

[0331] "Hello everyone. Today I'd like to talk to you about our new project."

[0332] Step 6:

[0333] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the original audio data. This module takes an audio file as input and outputs emotional data, which is reflected in the tone and pitch of the voice.

[0334] Step 7:

[0335] The generated audio data is assigned an identifier (e.g., NFT). The server generates the identifier using blockchain technology (e.g., Ethereum) and associates it with the audio data. The input is an audio file, and the output is an audio file with an identifier.

[0336] Step 8:

[0337] The server stores the voice data with the identifier in a database. The voice files with the identifier are stored in a database, and the database manages the voice data and maintains a transaction history.

[0338] Step 9:

[0339] The server publishes listing information for audio data sold by users on the marketplace. The input is an audio file with an identifier and a selling price, and the output is product information published on the marketplace. The server actually updates the interface to display seller information and audio previews.

[0340] Step 10:

[0341] When a user purchases audio data from the marketplace, the audio data is downloaded to the user's device. The input is a purchase instruction and an audio file with an identifier, and the output is the audio file saved on the user's device. Specifically, the download begins automatically once the purchase process is complete.

[0342] Step 11:

[0343] The device plays the downloaded audio data using the application's player function. The input is the downloaded audio file, and the output is the played audio. The audio starts playing when the user presses the play button in the app.

[0344] Step 12:

[0345] The user's device saves downloaded audio data to local storage and periodically backs it up to the cloud. The input is the audio file in local storage, and the output is the audio file stored in the cloud. Specifically, if the automatic backup setting is enabled, the data will be uploaded to cloud storage at the specified time.

[0346] (Application example 2)

[0347] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0348] Conventional voice generation services analyze user-recorded voice data and lack naturalness and diversity when using the generated voice data. Furthermore, there are no systems that can effectively respond to situations requiring real-time voice responses that reflect the user's emotions. This leads to insufficient customer service in virtual stores and a poor user experience.

[0349] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user voice data and acquiring text data and emotion data, means for generating natural voice data from the text data and emotion data, and means for assigning a unique identifier to the generated voice data and storing it in a database. This makes it possible to generate natural and diverse voice data that reflects emotions, and realizes real-time user responses in virtual stores, etc.

[0350] "Means for recording the user's voice" refers to a device or software for recording the user's spoken voice as digital data.

[0351] "Means for transmitting recorded voice data to a server" refers to a device or software that has the function of transferring recorded voice data to a server via a network.

[0352] "Means for analyzing voice data received by the server and converting it into text data" refers to technology for processing voice data received by the server and converting it into text in natural language.

[0353] "Means for obtaining emotional data from analyzed text data" refers to technology that uses analyzed text data to recognize and determine the emotional state of the speaker.

[0354] "Means for generating voice data from text data and emotional data" refers to technology that creates natural, emotionally rich voice data based on text data and emotional data.

[0355] "Means for assigning an identifier to generated voice data" refers to a technology for adding unique identification information to generated voice data to prove the authenticity and uniqueness of the data.

[0356] "Means for storing voice data with identifiers in a database" refers to a technology for securely storing voice data with unique identification information added in a database.

[0357] "Means for listing voice data with identifiers on a marketplace" refers to technology that makes voice data with identifiers attached publicly available on an online platform and makes it available for sale or trading.

[0358] "Means for managing transactions of voice data in a marketplace" refers to technology that oversees the buying, selling, and exchange of voice data on a marketplace and ensures that transactions are carried out smoothly.

[0359] "Means for downloading audio data purchased by a user to a terminal" refers to technology for saving audio data purchased by a user on a marketplace to the user's device.

[0360] "Means for playing downloaded audio data on a terminal" refers to software or hardware for playing audio data downloaded to a terminal.

[0361] "Means for generating and instantly playing back audio data in real time" refers to technology for instantly generating audio data based on audio or text input by a user and playing it back in real time.

[0362] The system for implementing this invention comprises a voice recording means, a server communication means, a voice analysis and generation means, an identifier assignment means, a database storage means, a marketplace management means, a download means, a playback means, a real-time voice generation means, and an emotion analysis means.

[0363] User processing

[0364] The user launches a dedicated application using a smartphone or head-mounted display (HMD). The application provides "Start Recording" and "Stop Recording" buttons. The user operates these buttons to record audio, and then presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[0365] Server-side processing

[0366] The server first temporarily stores the received voice data. It then converts the voice data into text data using the Google Cloud Speech-to-Text API. At the same time, it analyzes emotion data from the text data using the Emotion API. It then uses the Google Cloud Text-to-Speech API to generate natural-sounding voice data based on the text data and emotion data. The generated voice data is assigned an identifier using an NFT (non-fungible token) and stored in a database.

[0367] Marketplace operations

[0368] The server also includes a function for listing voice data with identifiers on the marketplace. Users can access the marketplace and list or purchase voice data. The server is also responsible for setting prices and managing transactions for voice data on the marketplace. Users who purchase voice data can download the voice data to their own devices via the cloud service.

[0369] Terminal side processing

[0370] The downloaded voice data can be played back on the user's device within a dedicated application. It is also possible to generate voice based on text data entered in real time and play it back instantly. By providing natural, rich voice responses that reflect emotion analysis, customer service in virtual stores is enhanced.

[0371] Examples of concrete examples and prompts

[0372] For example, consider the case where a user inquires about trying on clothes in a virtual store. When the user says, "Do you prefer this outfit in black or red?", the voice is recorded and sent to the server. The server analyzes the voice and obtains text data and emotion data. A voice response generated based on this data, "Black clothes are chic and lovely," is sent to the user's device and played back.

[0373] Example prompt for generative AI model:

[0374] "Write a Python program that converts the user's voice data into text, analyzes the emotion data, and then generates the speech again. Use the Google Cloud Speech-to-Text and Google Cloud Text-to-Speech APIs, and the Emotion API for emotion analysis."

[0375] This invention makes it possible to generate natural and diverse voice data that reflects emotions, thereby enhancing customer service in virtual stores, etc. This is expected to improve the user experience.

[0376] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0377] Step 1:

[0378] The user starts the dedicated application and clicks the "Start Recording" button to record audio.

[0379] Input: User utterance

[0380] Output: Recorded audio data

[0381] Specific operation: Records the user's voice using the microphone built into the smartphone or head-mounted display (HMD) and saves the audio file in local storage.

[0382] Step 2:

[0383] The user clicks the "End Recording" button and then presses the "Send" button to send the recorded voice data to the server.

[0384] Input: Recorded audio data

[0385] Output: Audio data sent to the server

[0386] Specific behavior: A process is executed to upload an audio file to a server via the Internet.

[0387] Step 3:

[0388] The server temporarily stores the received audio data.

[0389] Input: Audio data sent to the server

[0390] Output: Temporarily saved audio data

[0391] What it does: Saves the audio file to the server's local storage or temporary cache area.

[0392] Step 4:

[0393] The server uses the Google Cloud Speech-to-Text API to analyze the audio data and convert it into text data.

[0394] Input: Temporarily stored audio data

[0395] Output: Parsed text data

[0396] Specific operation: Sends an audio file to the API and retrieves the resulting string data as text data.

[0397] Step 5:

[0398] The server uses the Emotion API to obtain emotion data from text data.

[0399] Input: Parsed text data

[0400] Output: Emotion data

[0401] What it does: Sends text data to the Emotion API and retrieves data indicating the emotional state based on that text.

[0402] Step 6:

[0403] The server uses the Google Cloud Text-to-Speech API to generate voice data based on the text data and emotion data.

[0404] Input: Text data and emotion data

[0405] Output: Generated audio data

[0406] What it does: Sends text and emotion data to the API and retrieves the generated audio file.

[0407] Step 7:

[0408] The server assigns an NFT identifier to the generated audio data and stores it in a database.

[0409] Input: Generated audio data

[0410] Output: Audio data with identifiers

[0411] What it does: Adds a unique identifier to the audio file and stores it in a database.

[0412] Step 8:

[0413] The server puts the voice data with the identifier on the marketplace.

[0414] Input: Audio data with identifiers

[0415] Output: Audio data listed on the marketplace

[0416] What it does: Publishes audio files through the marketplace's management system, making them accessible to other users.

[0417] Step 9:

[0418] When a user purchases audio data from the marketplace, the data is downloaded to the device.

[0419] Input: Audio data selected from the marketplace

[0420] Output: Audio data downloaded to the device

[0421] Specific operation: Transfer data from cloud storage to the user's device and save it in local storage.

[0422] Step 10:

[0423] The terminal plays the downloaded audio data.

[0424] Input: Audio data downloaded to the device

[0425] Output: The audio data to be played

[0426] Specific Behavior: Plays an audio file using the audio player functionality within the application.

[0427] Step 11:

[0428] The device generates audio based on the text data entered in real time and plays it back instantly.

[0429] Input: Text data entered by the user

[0430] Output: Audio data generated and played in real time

[0431] Specific operation: Sends text data to the server, receives real-time generated audio data, and plays it back immediately.

[0432] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0433] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0434] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0435] [Second embodiment]

[0436] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0437] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0438] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0439] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0440] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0441] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0442] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0443] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0444] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0445] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0446] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0447] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0448] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data. The system of the present invention solves the problem by having the user, server, and terminal work together as shown below.

[0449] User processing

[0450] Audio Recording and Transmission

[0451] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[0452] Buying and selling

[0453] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[0454] Server-side processing

[0455] Speech Analysis and Generation

[0456] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[0457] Assigning identifiers

[0458] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[0459] Marketplace operations

[0460] The server publishes the audio data that users have listed on the marketplace. When a user purchases the data, the audio data is downloaded to the buyer's device. The server also manages transactions, notifies the buyer, and processes the seller's compensation.

[0461] Terminal side processing

[0462] Download and play audio

[0463] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[0464] Data Management and Backup

[0465] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[0466] Specific examples

[0467] As an example, consider a scenario where a user wants to record a narration for a presentation. The user uses a dedicated application to record the audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. Users can access a marketplace where they can sell their own audio data or purchase audio data from other users. The audio data is downloaded and played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[0468] By implementing these functions, the present invention solves the problems associated with users' voices and provides new value to voice data.

[0469] The processing flow will be explained below.

[0470] User processing steps

[0471] Step 1:

[0472] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[0473] Step 2:

[0474] The user presses the "Start Recording" button to start recording the audio, and when they have finished speaking, they press the "Stop Recording" button to stop recording.

[0475] Step 3:

[0476] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[0477] Step 4:

[0478] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[0479] Server-side processing steps

[0480] Step 1:

[0481] The server receives the voice data sent by the user and temporarily stores it.

[0482] Step 2:

[0483] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[0484] Step 3:

[0485] The server generates new voice data based on the text data using generative AI technology.

[0486] Step 4:

[0487] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[0488] Step 5:

[0489] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[0490] Step 6:

[0491] The server publishes the audio data that users have put up for sale on the marketplace and manages transactions of the audio data. When a transaction is completed, the audio data is provided to the buyer and a reward is paid to the seller.

[0492] Terminal processing steps

[0493] Step 1:

[0494] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[0495] Step 2:

[0496] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[0497] Step 3:

[0498] When a user inputs text data in real time, the terminal transmits this data to the server and instantly plays back the generated voice data.

[0499] Step 4:

[0500] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[0501] In this way, a system is realized in which the user, server, and terminal work in cooperation to effectively manage and utilize the user's voice data.

[0502] Example 1

[0503] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0504] Counterfeiting and unauthorized use of voice data have become a problem in the current voice data trading market. Conventional technologies lack a method to guarantee the uniqueness and authenticity of data when users record and trade their own voice data. This reduces the reliability of voice data and poses a challenge to ensuring transparency in transactions. Furthermore, the user experience is insufficient in the generation and management of voice data itself, as well as in post-purchase usage.

[0505] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0506] In this invention, the server includes means for recording the user's voice, means for transmitting the recorded voice data to the data server, means for analyzing the voice data received by the data server and converting it into text data, means for generating voice data from the text data, means for assigning an identifier to the generated voice data, means for storing the identified voice data in a database, means for listing the identified voice data on an online market, means for managing transactions of voice data on the online market, means for downloading voice data purchased by the user to an end-user terminal, and means for playing the voice data. This ensures the uniqueness and authenticity of the voice data and enables transparent transactions. Furthermore, users can efficiently generate, manage, trade, and use voice data.

[0507] "User" refers to any person or entity that uses the System to record voice and make transactions or purchases.

[0508] "Audio data" refers to data that represents audio information recorded by a user in digital form.

[0509] "Data Server" refers to a computer system that receives, analyzes, and stores voice data sent by users and manages transactions of voice data.

[0510] "Analysis" refers to the process of converting received voice data into text data.

[0511] "Text data" refers to character information obtained by analyzing voice data.

[0512] "Speech generation" refers to the process of generating new voice data based on the text data obtained through analysis.

[0513] "Identifier" refers to a digital tag or code that is attached to generated audio data to ensure the uniqueness and authenticity of the audio data.

[0514] "Database" refers to a storage device or storage system for storing identified audio data.

[0515] "Electronic Marketplace" refers to an online platform for trading identified voice data.

[0516] "End-user device" refers to a user's device (e.g., smartphone, personal computer) used to record, transmit, download, or play audio data.

[0517] "Playback" refers to the process of outputting audio data downloaded to an end-user terminal as an acoustic signal.

[0518] MODE FOR CARRYING OUT THE INVENTION

[0519] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data.

[0520] System Overview

[0521] The system uses the following hardware and software:

[0522] The device used by the user (e.g. smartphone, PC)

[0523] Dedicated application for recording and playback

[0524] Data server (receives, analyzes, and generates voice data)

[0525] Speech recognition technology for speech analysis (e.g., Google Cloud Speech-to-Text API)

[0526] AI models for speech generation (e.g., OpenAI GPT-3)

[0527] Database (storing voice data and identifiers)

[0528] Electronic market (voice data trading platform)

[0529] User processing

[0530] The user first launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, and the user presses the "Start Recording" button to begin recording. When recording is finished, the user presses the "Stop Recording" button, and then the "Send" button to upload the audio data to the server. At this time, the data is automatically sent via the Internet.

[0531] Server-side processing

[0532] The server temporarily stores the received voice data and performs voice analysis. The Google Cloud Speech-to-Text API is used to convert the voice data into text. The text data is then input into an AI model such as OpenAI GPT-3 to generate new voice data. This generated voice data is given an identifier such as an NFT to prevent counterfeiting or fraudulent use, and is stored in a database.

[0533] Terminal side processing

[0534] When a user purchases audio data from an online marketplace, the data is downloaded to the user's device via the Internet. The downloaded audio data is played using the player function within a dedicated application. Real-time audio generation is also possible, and audio data is instantly generated and played based on text data entered by the user.

[0535] Specific examples

[0536] As an example, consider a scenario in which a user wants to record narration for a presentation. The user launches a dedicated application and presses the "Start Recording" button to record the narration. When the recording is complete, the user presses the "Stop Recording" and "Send" buttons to send the audio data to the server. The server analyzes the audio data, converts it into text data, and then generates new audio data. This audio data is assigned an identifier and stored in a database. The user can then list the audio data on an electronic marketplace.

[0537] Prompt Sentence Examples

[0538] Please provide detailed instructions on how users can use the dedicated application to record audio and send the audio data to the server. Please also include details of the interface and the names of the operation buttons.

[0539] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0540] Step 1:

[0541] The user launches a dedicated application. The application screen displays a "Start Recording" button and a "Stop Recording" button. When the user presses the "Start Recording" button, the device's microphone is enabled and audio recording begins. The input is the user's voice, and the output is the recorded audio data.

[0542] Step 2:

[0543] When the user presses the "Stop Recording" button to end recording, the audio recording stops. The device temporarily stores this recording data in local storage. The input is an instruction to stop audio recording, and the output is the temporarily stored audio data.

[0544] Step 3:

[0545] The user presses the "Send" button to send the recorded data to the server. The device then uploads the audio data to the server via the Internet. The input is the temporarily saved audio data, and the output is the audio data sent to the server.

[0546] Step 4:

[0547] The server stores the received audio data in a temporary storage area. At this point, it performs a data integrity check to ensure that it is not corrupted. The input is the transmitted audio data, and the output is the audio data stored in the temporary storage area.

[0548] Step 5:

[0549] The server starts the speech analysis. Specifically, it converts the speech data into text data using the Google Cloud Speech-to-Text API. The input is the speech data in the temporary storage area, and the output is the analyzed text data.

[0550] Step 6:

[0551] The server inputs the text data into an AI speech generation model (e.g., OpenAI GPT-3) to generate new speech data. The input is the analyzed text data, and the output is the generated speech data.

[0552] Step 7:

[0553] The server assigns an identifier such as an NFT to the generated audio data. This procedure guarantees the uniqueness and authenticity of the audio data. The input is the generated audio data, and the output is audio data with an identifier.

[0554] Step 8:

[0555] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database.

[0556] Step 9:

[0557] Users access the marketplace to sell their own voice data or purchase other users' voice data. The marketplace provides an interface for managing transactions and setting prices. The input is a marketplace access instruction, and the output is transaction information.

[0558] Step 10:

[0559] When a user purchases audio data, the server downloads the audio data to the end-user terminal. The input is a purchase instruction, and the output is the audio data downloaded to the end-user terminal.

[0560] Step 11:

[0561] The device plays the downloaded audio data within a dedicated application. Pressing the play button starts audio playback. The input is the downloaded audio data, and the output is the played audio.

[0562] (Application example 1)

[0563] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0564] Conventional voice authentication systems have had problems with the accuracy and security of voice data authentication, and are not resistant to fraudulent use or attacks. Furthermore, there is a lack of transparency in the management and trading of voice authentication data, making it difficult to ensure trust between users. Furthermore, in addition to voice authentication, marketplace functions to increase the added value of voice data are lacking.

[0565] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0566] In this invention, the server includes a means for recording the user's voice, a means for transmitting the recorded voice data to the server, and a means for analyzing the received voice data and converting it into text data. This improves the accuracy of voice data authentication, prevents fraudulent use, and enables safe and efficient trading of voice data through the marketplace.

[0567] A "user" is an entity that uses the system and records, transmits, and plays back audio data.

[0568] "Voice data" is digital data that is a recording of a user's speech.

[0569] A "server" is a computer system that receives, analyzes, generates, identifies, stores, and manages voice data via a network.

[0570] "Analysis" is the process of processing received voice data and converting it into text data.

[0571] "Text data" is character information converted from audio data through analysis.

[0572] An "identifier" is unique information that is assigned to generated voice data to prevent counterfeiting or unauthorized use of the voice data.

[0573] "Marketplace" means an online platform where audio data is traded.

[0574] "Authentication" is the process of verifying a user's authenticity using recorded voice data.

[0575] "Matching" is the process of comparing generated and recorded audio data to confirm a match.

[0576] "Downloading" is the act of transferring audio data from a server to a user's terminal.

[0577] "Playback" refers to the act of outputting downloaded audio data as sound again on the terminal.

[0578] A system for implementing this invention requires the following components to record, transmit, analyze, generate, identify, store, and transact user voice data:

[0579] 1. User-side processing

[0580] Hardware and software:

[0581] Smartphone: Microphone, Storage

[0582] Dedicated applications: voice recording, transmission, player, etc.

[0583] First, users launch the dedicated application installed on their smartphone and tap the recording start button to record their own voice. When recording is finished, they press the send button to send the recorded voice data to the server. The voice data is automatically uploaded to the server via the Internet. Users can also access the marketplace to list their own voice data or purchase other voice data.

[0584] 2. Server-side processing

[0585] Hardware and software:

[0586] Cloud server: Database, speech analysis system, speech generation system, identifier assignment system, etc.

[0587] Google Cloud Speech-to-Text API: An analysis system for converting voice data into text data.

[0588] AI model: A system for generating new voice data from text data

[0589] The server receives the voice data sent by the user and temporarily stores it. The voice analysis system then converts the received voice data into text data. The text data is then input back into the voice generation system, and new voice data is generated using AI technology. A unique identifier is assigned to the generated voice data using an identifier assignment system. The voice data with the identifier assigned is stored in a database and made public on the marketplace. The marketplace manages downloads and transactions of voice data purchased by users.

[0590] 3. Processing on the user terminal side

[0591] Hardware and software:

[0592] Smartphone: Storage, player functions

[0593] Cloud service: Regular backup function

[0594] When a user purchases audio data from the marketplace, the audio data is downloaded to their smartphone. The downloaded audio data is played using the player function within the application. The audio data is also saved in local storage as needed and periodically backed up to the cloud.

[0595] Specific examples

[0596] For example, when a user wants to unlock their smartphone, they record their voice using a dedicated application, and after recording is complete, send it to the server. The server analyzes the voice and compares it with the newly generated voice data for authentication. At this time, the voice data with an identifier is stored in a database, preventing unauthorized use. If authentication is successful, the smartphone is unlocked.

[0597] Prompt Sentence Examples

[0598] Describe a scenario in which a user attempts to unlock a smartphone using a specific voice. Explain in detail the process from recording the user's voice and sending it to the server, to analyzing the voice on the server side, assigning an identifier, and finally multi-factor authentication.

[0599] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0600] Step 1:

[0601] The user records audio

[0602] The user launches a dedicated application on their smartphone and taps the "Start Recording" button to record their own voice. When the recording is complete, they press the "Stop Recording" button. In this step, the voice input from the user's microphone is saved as digital voice data. Specifically, the input data is the user's voice, and the output is an audio file saved on the smartphone.

[0603] Step 2:

[0604] Sending audio data from the device to the server

[0605] The user taps the "Send" button to send the recorded audio data to the server. The smartphone application uploads the audio data to the server as an HTTP POST request. The input is the recorded audio data, and the output is the audio data saved on the server. Specifically, the application saves the audio file as a temporary file and sends it to the server via the Internet.

[0606] Step 3:

[0607] The server analyzes the voice data and converts it into text data.

[0608] The server analyzes the received voice data and converts it into text data using a voice analysis system (e.g., Google Cloud Speech-to-Text API). The input is voice data, and the output is the converted text data. Specifically, the server sends the voice data to the API and receives the converted text.

[0609] Step 4:

[0610] The server generates new voice data from the text data.

[0611] The server inputs the text data into a speech generation system to generate new speech data. A generative AI model is used here. The input is text data, and the output is newly generated speech data. Specifically, the AI ​​speech generation model analyzes the text data and generates a speech file.

[0612] Step 5:

[0613] The server assigns an identifier to the generated audio data.

[0614] The server uses an identification system to assign a unique identifier (e.g., NFT) to the generated audio data. The input is the generated audio data, and the output is audio data with an identifier. Specifically, the audio data is processed using a hash function or similar to generate the identifier.

[0615] Step 6:

[0616] Save the identified audio data in a database

[0617] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database. Specifically, the voice data is stored in a fixed format using a database management system.

[0618] Step 7:

[0619] Publishing audio data with identifiers to the marketplace

[0620] The server publishes the identified voice data to the marketplace, allowing users to purchase and trade. The input is the voice data stored in the database, and the output is the voice data posted on the marketplace. Specifically, the web server displays the voice data through the marketplace interface.

[0621] Step 8:

[0622] The user purchases the audio data and downloads it to their device.

[0623] Users purchase audio data from the marketplace and download it to their smartphone. The input is the audio data on the marketplace, and the output is the audio data stored on the smartphone. Specifically, the application downloads the audio data using an HTTP request and saves it in local storage.

[0624] Step 9:

[0625] Play audio data on the device

[0626] The user plays the downloaded audio data using the player function of the dedicated application. The input is the audio data stored in local storage, and the output is the played audio. Specifically, the application's audio player reads the audio data and sends it to the audio output device (speakers or earphones) for playback.

[0627] This enables users to perform secure voice authentication for multi-stage authentication, and enables safe and efficient transactions of voice data with identifiers.

[0628] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0629] The present invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and then stores and trades it in a database, and further combines it with an emotion engine to increase the naturalness and diversity of the voice data. The system of the present invention solves these problems by having the user, server, and terminal work together.

[0630] User processing

[0631] Audio Recording and Transmission

[0632] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[0633] Buying and selling

[0634] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[0635] Server-side processing

[0636] Speech Analysis and Generation

[0637] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[0638] Combining Emotion Engines

[0639] In this invention, an emotion engine is built into the server, which can recognize emotions from the user's voice data. The emotion engine acquires the user's emotion data and reflects it in the generated voice data. For example, if the user is speaking with a cheerful tone, that emotion can be reflected in the tone and pitch of the voice data.

[0640] Assigning identifiers

[0641] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[0642] Marketplace operations

[0643] The server publishes voice data that users have listed on the marketplace. When a user purchases a voice data item, the voice data is downloaded to the buyer's device. The server also manages transactions, notifies buyers, and processes sellers' compensation. It can also automatically categorize voice data using emotion data recognized by the emotion engine.

[0644] Terminal side processing

[0645] Download and play audio

[0646] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[0647] Data Management and Backup

[0648] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[0649] Specific examples

[0650] As an example, consider a scenario in which a user is recording narration for a presentation. The user uses a dedicated application to record audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. An emotion engine is also used to recognize the user's emotions and reflect them in the generated audio data. This data is then put up for sale on a marketplace where other users can purchase it. The audio data downloaded by the purchaser is then played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[0651] By implementing these functions, the present invention solves the problems of users' voices and provides new value to voice data. By combining it with an emotion engine, it becomes possible to generate even more natural and emotionally rich voice data.

[0652] The processing flow will be explained below.

[0653] User processing steps

[0654] Step 1:

[0655] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[0656] Step 2:

[0657] The user presses the "Start Recording" button to begin recording the audio.

[0658] Step 3:

[0659] When the user has finished speaking, he or she presses the "End Recording" button to stop recording.

[0660] Step 4:

[0661] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[0662] Step 5:

[0663] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[0664] ---

[0665] Server-side processing steps

[0666] Step 1:

[0667] The server receives the voice data sent by the user and temporarily stores it.

[0668] Step 2:

[0669] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[0670] Step 3:

[0671] The server uses generative AI technology to generate new voice data based on the text data, and is equipped with an emotion engine that recognizes the user's emotions from the voice data.

[0672] Step 4:

[0673] The emotion engine analyzes the recognized emotion data and reflects it in the generated voice data. For example, if the user is speaking with a happy expression, that emotion will be reflected in the tone and pitch of the voice data.

[0674] Step 5:

[0675] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[0676] Step 6:

[0677] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[0678] Step 7:

[0679] The server publishes the voice data that users have put up for sale on the marketplace and manages the transactions of the voice data. When a transaction is completed, the server provides the voice data to the buyer and pays a fee to the seller. The server also automatically categorizes the voice data using emotion data recognized by the emotion engine.

[0680] ---

[0681] Terminal processing steps

[0682] Step 1:

[0683] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[0684] Step 2:

[0685] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[0686] Step 3:

[0687] When a user inputs text data in real time, the device transmits the data to the server and instantly plays back the generated voice data, with the emotion engine reflecting appropriate emotions in the voice data based on the text data entered by the user.

[0688] Step 4:

[0689] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[0690] In this way, a system is realized in which the user, server, and terminal work together to effectively manage and utilize the user's voice data. In addition, the introduction of an emotion engine greatly improves the naturalness and emotional impact of the voice data.

[0691] Example 2

[0692] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0693] The present invention aims to generate more natural and diverse voice data by reflecting emotions when analyzing and generating voice data. Another objective of the present invention is to improve the reliability and convenience of voice data by realizing secure transaction management of voice data with assigned identifiers and cloud backup of voice data.

[0694] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing emotions from user voice data and reflecting the emotions in generated voice data, a means for saving multiple voice data input by the user in local storage and backing them up to the cloud as needed, and a means for managing transactions of voice data in the marketplace. This makes it possible to generate more natural and emotional voice data that reflects emotions, and also improve the reliability and safety of the voice data.

[0695] "User" refers to any individual or entity that records, transmits, buys, or sells audio data.

[0696] "Server" refers to a central management system for analyzing, generating, and managing voice data sent by users.

[0697] "Terminal" refers to the device used by a user (e.g., smartphone, tablet, computer, etc.).

[0698] "Audio Data" refers to digital files that contain recordings of a user's voice.

[0699] "Text data" refers to character information that has been converted from analyzed voice data.

[0700] "Identifier" refers to a unique identification number or NFT assigned to audio data.

[0701] "Marketplace" refers to an online platform for trading voice data.

[0702] An "emotion engine" refers to a system that recognizes a user's emotions from voice data and reflects them in the generated voice data.

[0703] "Local storage" refers to a storage area for storing data within a terminal.

[0704] "Cloud" refers to data storage provided on remote servers accessible over the Internet.

[0705] A "generative AI model" refers to an artificial intelligence algorithm for generating new voice data based on text data.

[0706] This invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and stores and trades it in a database. The main purpose is to increase the naturalness and diversity of voice data by combining it with an emotion engine. The system operates in cooperation with the user, server, and terminal.

[0707] User processing

[0708] Audio Recording and Transmission

[0709] The user starts the dedicated application and starts recording audio. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. This transmission process is carried out automatically via the Internet.

[0710] Buying and selling

[0711] Users can access the marketplace and list their own audio data or purchase audio data listed by other users. The marketplace provides an interface for setting the selling price. For example, a user can record a narration for a presentation and list it for 500 yen. The following is an example of a prompt:

[0712] "Hello everyone. Today I'd like to talk to you about our new project."

[0713] Server-side processing

[0714] Speech Analysis and Generation

[0715] The server receives the voice data sent by the user and temporarily stores it. It then uses Google Cloud's speech recognition API to analyze the voice data and convert it into text data. This text data is then input back into a speech generation system (e.g., a generative AI model) to generate new voice data. A generative AI model such as OpenAI's GPT-3 could be used.

[0716] Combining Emotion Engines

[0717] The server is equipped with an emotion engine that recognizes emotions from the user's voice data. It acquires emotion data using tools such as IBM Watson Tone Analyzer and reflects that emotion in the generated voice data. For example, if the user sounds like they're having fun, that emotion can be reflected in the tone and pitch of the voice data.

[0718] Assigning identifiers

[0719] The generated voice data is assigned an identifier, such as an NFT (Non-Fungible Token). The identifiers are managed using the Ethereum blockchain or similar to prove that the voice data is unique. After the identifier is assigned, the voice data is stored in a database.

[0720] Marketplace operations

[0721] The server publishes the voice data that users have put up for sale on the marketplace. When a user purchases voice data, the server downloads the data to the buyer's device, manages the transaction, notifies the buyer, and processes compensation for the seller. It also automatically sets the category of the voice data using data from the emotion engine.

[0722] Terminal side processing

[0723] Download and play audio

[0724] When a user purchases audio data from the marketplace, it is automatically downloaded to the device and can be played using the player function within the application. It is also possible to generate audio based on text data entered in real time and play it instantly.

[0725] Data Management and Backup

[0726] The user's device will save downloaded audio data to local storage as needed. In addition, important audio data will be backed up to the cloud periodically. Backups can be automated using the Google Drive API, for example.

[0727] Specific examples

[0728] For example, consider a scenario where a user is recording narration for a presentation. The user uses a dedicated app to record audio and send it to the server. The server then uses Google Cloud's speech recognition API to convert the audio data into text and generates new audio data using the OpenAI GPT-3 model. This audio data is assigned an NFT identifier using the Ethereum blockchain. An emotion engine is also used to recognize emotions and reflect them in tone and pitch. This data is then put up for sale on a marketplace where other users can purchase it. Buyers can play the downloaded audio data using the player function. Real-time audio generation is also possible, allowing for audio substitution even if the user is unable to speak.

[0729] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0730] Step 1:

[0731] The user launches a dedicated application. They press the "Start Recording" button in the application and then speak into the microphone to record audio data. When recording is complete, they press the "Stop Recording" button. The input is the user's voice, and the output is an audio file (e.g., WAV format). Specifically, the user taps a button on the app screen on their smartphone.

[0732] Step 2:

[0733] The user presses the "Send" button to send the recorded audio data to the server. The data is automatically uploaded to the server via the Internet. The input is the recorded audio file, and the output is the audio file saved on the server. During the sending process, a progress bar is displayed on the screen.

[0734] Step 3:

[0735] The server temporarily stores the received voice data. The stored voice file is used as input data for analysis processing. Specifically, the received data is stored in a temporary folder.

[0736] Step 4:

[0737] The server converts the voice data into text data using Google Cloud's speech recognition API. Here, the voice file is input and the converted text data is output. A specific API call is made to convert the voice data into text information.

[0738] Step 5:

[0739] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate new voice data from text data. The input is text data, and the output is a new voice file. In this case, the voice data is generated by giving a prompt to the AI ​​model. For example, the following prompt sentence is used:

[0740] "Hello everyone. Today I'd like to talk to you about our new project."

[0741] Step 6:

[0742] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the original audio data. This module takes an audio file as input and outputs emotional data, which is reflected in the tone and pitch of the voice.

[0743] Step 7:

[0744] The generated audio data is assigned an identifier (e.g., NFT). The server generates the identifier using blockchain technology (e.g., Ethereum) and associates it with the audio data. The input is an audio file, and the output is an audio file with an identifier.

[0745] Step 8:

[0746] The server stores the voice data with the identifier in a database. The voice files with the identifier are stored in a database, and the database manages the voice data and maintains a transaction history.

[0747] Step 9:

[0748] The server publishes listing information for audio data sold by users on the marketplace. The input is an audio file with an identifier and a selling price, and the output is the product information published on the marketplace. The server actually updates the interface to display seller information and a preview of the audio.

[0749] Step 10:

[0750] When a user purchases audio data from the marketplace, the audio data is downloaded to the user's device. The input is a purchase instruction and an audio file with an identifier, and the output is the audio file saved on the user's device. Specifically, the download begins automatically once the purchase process is complete.

[0751] Step 11:

[0752] The device plays the downloaded audio data using the application's player function. The input is the downloaded audio file, and the output is the played audio. The audio starts playing when the user presses the play button in the app.

[0753] Step 12:

[0754] The user's device saves downloaded audio data to local storage and periodically backs it up to the cloud. The input is the audio file in local storage, and the output is the audio file stored in the cloud. Specifically, if the automatic backup setting is enabled, the data will be uploaded to cloud storage at the specified time.

[0755] (Application example 2)

[0756] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0757] Conventional voice generation services analyze user-recorded voice data and lack naturalness and diversity when using the generated voice data. Furthermore, there are no systems that can effectively respond to situations requiring real-time voice responses that reflect the user's emotions. This leads to insufficient customer service in virtual stores and a poor user experience.

[0758] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user voice data and acquiring text data and emotion data, means for generating natural voice data from the text data and emotion data, and means for assigning a unique identifier to the generated voice data and storing it in a database. This makes it possible to generate natural and diverse voice data that reflects emotions, and realizes real-time user responses in virtual stores, etc.

[0759] "Means for recording the user's voice" refers to a device or software for recording the user's spoken voice as digital data.

[0760] "Means for transmitting recorded voice data to a server" refers to a device or software that has the function of transferring recorded voice data to a server via a network.

[0761] "Means for analyzing voice data received by the server and converting it into text data" refers to technology for processing voice data received by the server and converting it into text in natural language.

[0762] "Means for obtaining emotional data from analyzed text data" refers to technology that uses analyzed text data to recognize and determine the emotional state of the speaker.

[0763] "Means for generating voice data from text data and emotional data" refers to technology that creates natural, emotionally rich voice data based on text data and emotional data.

[0764] "Means for assigning an identifier to generated voice data" refers to a technology for adding unique identification information to generated voice data to prove the authenticity and uniqueness of the data.

[0765] "Means for storing voice data with identifiers in a database" refers to a technology for securely storing voice data with unique identification information added in a database.

[0766] "Means for listing voice data with identifiers on a marketplace" refers to technology that makes voice data with identifiers attached publicly available on an online platform and makes it available for sale or trading.

[0767] "Means for managing transactions of voice data in a marketplace" refers to technology that oversees the buying, selling, and exchange of voice data on a marketplace and ensures that transactions are carried out smoothly.

[0768] "Means for downloading audio data purchased by a user to a terminal" refers to technology for saving audio data purchased by a user on a marketplace to the user's device.

[0769] "Means for playing downloaded audio data on a terminal" refers to software or hardware for playing audio data downloaded to a terminal.

[0770] "Means for generating and instantly playing back audio data in real time" refers to technology for instantly generating audio data based on audio or text input by a user and playing it back in real time.

[0771] The system for implementing this invention comprises a voice recording means, a server communication means, a voice analysis and generation means, an identifier assignment means, a database storage means, a marketplace management means, a download means, a playback means, a real-time voice generation means, and an emotion analysis means.

[0772] User processing

[0773] The user launches a dedicated application using a smartphone or head-mounted display (HMD). The application provides "Start Recording" and "Stop Recording" buttons. The user operates these buttons to record audio, and then presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[0774] Server-side processing

[0775] The server first temporarily stores the received voice data. It then converts the voice data into text data using the Google Cloud Speech-to-Text API. At the same time, it analyzes emotion data from the text data using the Emotion API. It then uses the Google Cloud Text-to-Speech API to generate natural-sounding voice data based on the text data and emotion data. The generated voice data is assigned an identifier using an NFT (non-fungible token) and stored in a database.

[0776] Marketplace operations

[0777] The server also includes a function for listing voice data with identifiers on the marketplace. Users can access the marketplace and list or purchase voice data. The server is also responsible for setting prices and managing transactions for voice data on the marketplace. Users who purchase voice data can download the voice data to their own devices via the cloud service.

[0778] Terminal side processing

[0779] The downloaded voice data can be played back on the user's device within a dedicated application. It is also possible to generate voice based on text data entered in real time and play it back instantly. By providing natural, rich voice responses that reflect emotion analysis, customer service in virtual stores is enhanced.

[0780] Examples of concrete examples and prompts

[0781] For example, consider the case where a user inquires about trying on clothes in a virtual store. When the user says, "Do you prefer this outfit in black or red?", the voice is recorded and sent to the server. The server analyzes the voice and obtains text data and emotion data. A voice response generated based on this data, "Black clothes are chic and lovely," is sent to the user's device and played back.

[0782] Example prompt for generative AI model:

[0783] "Write a Python program that converts the user's voice data into text, analyzes the emotion data, and then generates the speech again. Use the Google Cloud Speech-to-Text and Google Cloud Text-to-Speech APIs, and the Emotion API for emotion analysis."

[0784] This invention makes it possible to generate natural and diverse voice data that reflects emotions, thereby enhancing customer service in virtual stores, etc. This is expected to improve the user experience.

[0785] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0786] Step 1:

[0787] The user starts the dedicated application and clicks the "Start Recording" button to record audio.

[0788] Input: User utterance

[0789] Output: Recorded audio data

[0790] Specific operation: Records the user's voice using the microphone built into the smartphone or head-mounted display (HMD) and saves the audio file in local storage.

[0791] Step 2:

[0792] The user clicks the "End Recording" button and then presses the "Send" button to send the recorded voice data to the server.

[0793] Input: Recorded audio data

[0794] Output: Audio data sent to the server

[0795] Specific behavior: A process is executed to upload an audio file to a server via the Internet.

[0796] Step 3:

[0797] The server temporarily stores the received audio data.

[0798] Input: Audio data sent to the server

[0799] Output: Temporarily saved audio data

[0800] What it does: Saves the audio file to the server's local storage or temporary cache area.

[0801] Step 4:

[0802] The server uses the Google Cloud Speech-to-Text API to analyze the audio data and convert it into text data.

[0803] Input: Temporarily stored audio data

[0804] Output: Parsed text data

[0805] Specific operation: Sends an audio file to the API and retrieves the resulting string data as text data.

[0806] Step 5:

[0807] The server uses the Emotion API to obtain emotion data from text data.

[0808] Input: Parsed text data

[0809] Output: Emotion data

[0810] What it does: Sends text data to the Emotion API and retrieves data indicating the emotional state based on that text.

[0811] Step 6:

[0812] The server uses the Google Cloud Text-to-Speech API to generate voice data based on the text data and emotion data.

[0813] Input: Text data and emotion data

[0814] Output: Generated audio data

[0815] What it does: Sends text and emotion data to the API and retrieves the generated audio file.

[0816] Step 7:

[0817] The server assigns an NFT identifier to the generated audio data and stores it in a database.

[0818] Input: Generated audio data

[0819] Output: Audio data with identifiers

[0820] What it does: Adds a unique identifier to the audio file and stores it in a database.

[0821] Step 8:

[0822] The server puts the voice data with the identifier on the marketplace.

[0823] Input: Audio data with identifiers

[0824] Output: Audio data listed on the marketplace

[0825] What it does: Publishes audio files through the marketplace's management system, making them accessible to other users.

[0826] Step 9:

[0827] When a user purchases audio data from the marketplace, the data is downloaded to the device.

[0828] Input: Audio data selected from the marketplace

[0829] Output: Audio data downloaded to the device

[0830] Specific operation: Transfer data from cloud storage to the user's device and save it in local storage.

[0831] Step 10:

[0832] The terminal plays the downloaded audio data.

[0833] Input: Audio data downloaded to the device

[0834] Output: The audio data to be played

[0835] Specific Behavior: Plays an audio file using the audio player functionality within the application.

[0836] Step 11:

[0837] The device generates audio based on the text data entered in real time and plays it back instantly.

[0838] Input: Text data entered by the user

[0839] Output: Audio data generated and played in real time

[0840] Specific operation: Sends text data to the server, receives real-time generated audio data, and plays it back immediately.

[0841] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0842] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0843] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0844] [Third embodiment]

[0845] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0846] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0847] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0848] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0849] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0850] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0851] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0852] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0853] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0854] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0855] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0856] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0857] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data. The system of the present invention solves the problem by having the user, server, and terminal work together as shown below.

[0858] User processing

[0859] Audio Recording and Transmission

[0860] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[0861] Buying and selling

[0862] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[0863] Server-side processing

[0864] Speech Analysis and Generation

[0865] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[0866] Assigning identifiers

[0867] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[0868] Marketplace operations

[0869] The server publishes the audio data that users have listed on the marketplace. When a user purchases the data, the audio data is downloaded to the buyer's device. The server also manages transactions, notifies the buyer, and processes the seller's compensation.

[0870] Terminal side processing

[0871] Download and play audio

[0872] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[0873] Data Management and Backup

[0874] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[0875] Specific examples

[0876] As an example, consider a scenario where a user wants to record a narration for a presentation. The user uses a dedicated application to record the audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. Users can access a marketplace where they can sell their own audio data or purchase audio data from other users. The audio data is downloaded and played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[0877] By implementing these functions, the present invention solves the problems associated with users' voices and provides new value to voice data.

[0878] The processing flow will be explained below.

[0879] User processing steps

[0880] Step 1:

[0881] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[0882] Step 2:

[0883] The user presses the "Start Recording" button to start recording the audio, and when they have finished speaking, they press the "Stop Recording" button to stop recording.

[0884] Step 3:

[0885] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[0886] Step 4:

[0887] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[0888] Server-side processing steps

[0889] Step 1:

[0890] The server receives the voice data sent by the user and temporarily stores it.

[0891] Step 2:

[0892] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[0893] Step 3:

[0894] The server generates new voice data based on the text data using generative AI technology.

[0895] Step 4:

[0896] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[0897] Step 5:

[0898] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[0899] Step 6:

[0900] The server publishes the audio data that users have put up for sale on the marketplace and manages transactions of the audio data. When a transaction is completed, the audio data is provided to the buyer and a fee is paid to the seller.

[0901] Terminal processing steps

[0902] Step 1:

[0903] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[0904] Step 2:

[0905] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[0906] Step 3:

[0907] When a user inputs text data in real time, the terminal transmits this data to the server and instantly plays back the generated voice data.

[0908] Step 4:

[0909] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[0910] In this way, a system is realized in which the user, server, and terminal work in cooperation to effectively manage and utilize the user's voice data.

[0911] Example 1

[0912] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0913] Counterfeiting and unauthorized use of voice data have become a problem in the current voice data trading market. Conventional technologies lack a method to guarantee the uniqueness and authenticity of data when users record and trade their own voice data. This reduces the reliability of voice data and poses a challenge to ensuring transparency in transactions. Furthermore, the user experience is insufficient in the generation and management of voice data itself, as well as in post-purchase usage.

[0914] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0915] In this invention, the server includes means for recording the user's voice, means for transmitting the recorded voice data to the data server, means for analyzing the voice data received by the data server and converting it into text data, means for generating voice data from the text data, means for assigning an identifier to the generated voice data, means for storing the identified voice data in a database, means for listing the identified voice data on an online market, means for managing transactions of voice data on the online market, means for downloading voice data purchased by the user to an end-user terminal, and means for playing the voice data. This ensures the uniqueness and authenticity of the voice data and enables transparent transactions. Furthermore, users can efficiently generate, manage, trade, and use voice data.

[0916] "User" refers to any person or entity that uses the System to record voice and make transactions or purchases.

[0917] "Audio data" refers to data that represents audio information recorded by a user in digital form.

[0918] "Data Server" refers to a computer system that receives, analyzes, and stores voice data sent by users and manages transactions of voice data.

[0919] "Analysis" refers to the process of converting received voice data into text data.

[0920] "Text data" refers to character information obtained by analyzing voice data.

[0921] "Speech generation" refers to the process of generating new voice data based on the text data obtained through analysis.

[0922] "Identifier" refers to a digital tag or code that is attached to generated audio data to ensure the uniqueness and authenticity of the audio data.

[0923] "Database" refers to a storage device or storage system for storing identified audio data.

[0924] "Electronic Marketplace" refers to an online platform for trading identified voice data.

[0925] "End-user device" refers to a user's device (e.g., smartphone, personal computer) used to record, transmit, download, or play audio data.

[0926] "Playback" refers to the process of outputting audio data downloaded to an end-user terminal as an acoustic signal.

[0927] MODE FOR CARRYING OUT THE INVENTION

[0928] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data.

[0929] System Overview

[0930] The system uses the following hardware and software:

[0931] The device used by the user (e.g. smartphone, PC)

[0932] Dedicated application for recording and playback

[0933] Data server (receives, analyzes, and generates voice data)

[0934] Speech recognition technology for speech analysis (e.g., Google Cloud Speech-to-Text API)

[0935] AI models for speech generation (e.g., OpenAI GPT-3)

[0936] Database (storing voice data and identifiers)

[0937] Electronic market (voice data trading platform)

[0938] User processing

[0939] The user first launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, and the user presses the "Start Recording" button to begin recording. When recording is finished, the user presses the "Stop Recording" button, and then the "Send" button to upload the audio data to the server. At this time, the data is automatically sent via the Internet.

[0940] Server-side processing

[0941] The server temporarily stores the received voice data and performs voice analysis. The Google Cloud Speech-to-Text API is used to convert the voice data into text. The text data is then input into an AI model such as OpenAI GPT-3 to generate new voice data. This generated voice data is given an identifier such as an NFT to prevent counterfeiting or fraudulent use, and is stored in a database.

[0942] Terminal side processing

[0943] When a user purchases audio data from an online marketplace, the data is downloaded to the user's device via the Internet. The downloaded audio data is played using the player function within a dedicated application. Real-time audio generation is also possible, and audio data is instantly generated and played based on text data entered by the user.

[0944] Specific examples

[0945] As an example, consider a scenario in which a user wants to record narration for a presentation. The user launches a dedicated application and presses the "Start Recording" button to record the narration. When the recording is complete, the user presses the "Stop Recording" and "Send" buttons to send the audio data to the server. The server analyzes the audio data, converts it into text data, and then generates new audio data. This audio data is assigned an identifier and stored in a database. The user can then list the audio data on an electronic marketplace.

[0946] Prompt Sentence Examples

[0947] Please provide detailed instructions on how users can use the dedicated application to record audio and send the audio data to the server. Please also include details of the interface and the names of the operation buttons.

[0948] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0949] Step 1:

[0950] The user launches a dedicated application. The application screen displays a "Start Recording" button and a "Stop Recording" button. When the user presses the "Start Recording" button, the device's microphone is enabled and audio recording begins. The input is the user's voice, and the output is the recorded audio data.

[0951] Step 2:

[0952] When the user presses the "Stop Recording" button to end recording, the audio recording stops. The device temporarily stores this recording data in local storage. The input is an instruction to stop audio recording, and the output is the temporarily stored audio data.

[0953] Step 3:

[0954] The user presses the "Send" button to send the recorded data to the server. The device then uploads the audio data to the server via the Internet. The input is the temporarily saved audio data, and the output is the audio data sent to the server.

[0955] Step 4:

[0956] The server stores the received audio data in a temporary storage area. At this point, it performs a data integrity check to ensure that it is not corrupted. The input is the transmitted audio data, and the output is the audio data stored in the temporary storage area.

[0957] Step 5:

[0958] The server starts the speech analysis. Specifically, it converts the speech data into text data using the Google Cloud Speech-to-Text API. The input is the speech data in the temporary storage area, and the output is the analyzed text data.

[0959] Step 6:

[0960] The server inputs the text data into an AI speech generation model (e.g., OpenAI GPT-3) to generate new speech data. The input is the analyzed text data, and the output is the generated speech data.

[0961] Step 7:

[0962] The server assigns an identifier such as an NFT to the generated audio data. This procedure guarantees the uniqueness and authenticity of the audio data. The input is the generated audio data, and the output is audio data with an identifier.

[0963] Step 8:

[0964] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database.

[0965] Step 9:

[0966] Users access the marketplace to sell their own voice data or purchase other users' voice data. The marketplace provides an interface for managing transactions and setting prices. The input is a marketplace access instruction, and the output is transaction information.

[0967] Step 10:

[0968] When a user purchases audio data, the server downloads the audio data to the end-user terminal. The input is a purchase instruction, and the output is the audio data downloaded to the end-user terminal.

[0969] Step 11:

[0970] The device plays the downloaded audio data within a dedicated application. Pressing the play button starts audio playback. The input is the downloaded audio data, and the output is the played audio.

[0971] (Application example 1)

[0972] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0973] Conventional voice authentication systems have had problems with the accuracy and security of voice data authentication, and are not resistant to fraudulent use or attacks. Furthermore, there is a lack of transparency in the management and trading of voice authentication data, making it difficult to ensure trust between users. Furthermore, in addition to voice authentication, marketplace functions to increase the added value of voice data are lacking.

[0974] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0975] In this invention, the server includes a means for recording the user's voice, a means for transmitting the recorded voice data to the server, and a means for analyzing the received voice data and converting it into text data. This improves the accuracy of voice data authentication, prevents fraudulent use, and enables safe and efficient trading of voice data through the marketplace.

[0976] A "user" is an entity that uses the system and records, transmits, and plays back audio data.

[0977] "Voice data" is digital data that is a recording of a user's speech.

[0978] A "server" is a computer system that receives, analyzes, generates, identifies, stores, and manages voice data via a network.

[0979] "Analysis" is the process of processing received voice data and converting it into text data.

[0980] "Text data" is character information converted from audio data through analysis.

[0981] An "identifier" is unique information that is assigned to generated voice data to prevent counterfeiting or unauthorized use of the voice data.

[0982] "Marketplace" means an online platform where audio data is traded.

[0983] "Authentication" is the process of verifying a user's authenticity using recorded voice data.

[0984] "Matching" is the process of comparing generated and recorded audio data to confirm a match.

[0985] "Downloading" is the act of transferring audio data from a server to a user's terminal.

[0986] "Playback" refers to the act of outputting downloaded audio data as sound again on the terminal.

[0987] A system for implementing this invention requires the following components to record, transmit, analyze, generate, identify, store, and transact user voice data:

[0988] 1. User-side processing

[0989] Hardware and software:

[0990] Smartphone: Microphone, Storage

[0991] Dedicated applications: voice recording, transmission, player, etc.

[0992] First, users launch the dedicated application installed on their smartphone and tap the recording start button to record their own voice. When recording is finished, they press the send button to send the recorded voice data to the server. The voice data is automatically uploaded to the server via the Internet. Users can also access the marketplace to list their own voice data or purchase other voice data.

[0993] 2. Server-side processing

[0994] Hardware and software:

[0995] Cloud server: Database, speech analysis system, speech generation system, identifier assignment system, etc.

[0996] Google Cloud Speech-to-Text API: An analysis system for converting voice data into text data.

[0997] AI model: A system for generating new voice data from text data

[0998] The server receives the voice data sent by the user and temporarily stores it. The voice analysis system then converts the received voice data into text data. The text data is then input back into the voice generation system, and new voice data is generated using AI technology. A unique identifier is assigned to the generated voice data using an identifier assignment system. The voice data with the identifier assigned is stored in a database and made public on the marketplace. The marketplace manages downloads and transactions of voice data purchased by users.

[0999] 3. Processing on the user terminal side

[1000] Hardware and software:

[1001] Smartphone: Storage, player functions

[1002] Cloud service: Regular backup function

[1003] When a user purchases audio data from the marketplace, the audio data is downloaded to their smartphone. The downloaded audio data is played using the player function within the application. The audio data is also saved in local storage as needed and periodically backed up to the cloud.

[1004] Specific examples

[1005] For example, when a user wants to unlock their smartphone, they record their voice using a dedicated application, and after recording is complete, send it to the server. The server analyzes the voice and compares it with the newly generated voice data for authentication. At this time, the voice data with an identifier is stored in a database, preventing unauthorized use. If authentication is successful, the smartphone is unlocked.

[1006] Prompt Sentence Examples

[1007] Describe a scenario in which a user attempts to unlock a smartphone using a specific voice. Explain in detail the process from recording the user's voice and sending it to the server, to analyzing the voice on the server side, assigning an identifier, and finally multi-factor authentication.

[1008] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1009] Step 1:

[1010] The user records audio

[1011] The user launches a dedicated application on their smartphone and taps the "Start Recording" button to record their own voice. When the recording is complete, they press the "Stop Recording" button. In this step, the voice input from the user's microphone is saved as digital voice data. Specifically, the input data is the user's voice, and the output is an audio file saved on the smartphone.

[1012] Step 2:

[1013] Sending audio data from the device to the server

[1014] The user taps the "Send" button to send the recorded audio data to the server. The smartphone application uploads the audio data to the server as an HTTP POST request. The input is the recorded audio data, and the output is the audio data saved on the server. Specifically, the application saves the audio file as a temporary file and sends it to the server via the Internet.

[1015] Step 3:

[1016] The server analyzes the voice data and converts it into text data.

[1017] The server analyzes the received voice data and converts it into text data using a voice analysis system (e.g., Google Cloud Speech-to-Text API). The input is voice data, and the output is the converted text data. Specifically, the server sends the voice data to the API and receives the converted text.

[1018] Step 4:

[1019] The server generates new voice data from the text data.

[1020] The server inputs the text data into a speech generation system to generate new speech data. A generative AI model is used here. The input is text data, and the output is newly generated speech data. Specifically, the AI ​​speech generation model analyzes the text data and generates a speech file.

[1021] Step 5:

[1022] The server assigns an identifier to the generated audio data.

[1023] The server uses an identification system to assign a unique identifier (e.g., NFT) to the generated audio data. The input is the generated audio data, and the output is audio data with an identifier. Specifically, the audio data is processed using a hash function or similar to generate the identifier.

[1024] Step 6:

[1025] Save the identified audio data in a database

[1026] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database. Specifically, the voice data is stored in a fixed format using a database management system.

[1027] Step 7:

[1028] Publishing audio data with identifiers to the marketplace

[1029] The server publishes the identified voice data to the marketplace, allowing users to purchase and trade. The input is the voice data stored in the database, and the output is the voice data posted on the marketplace. Specifically, the web server displays the voice data through the marketplace interface.

[1030] Step 8:

[1031] The user purchases the audio data and downloads it to their device.

[1032] Users purchase audio data from the marketplace and download it to their smartphone. The input is the audio data on the marketplace, and the output is the audio data stored on the smartphone. Specifically, the application downloads the audio data using an HTTP request and saves it in local storage.

[1033] Step 9:

[1034] Play audio data on the device

[1035] The user plays the downloaded audio data using the player function of the dedicated application. The input is the audio data stored in local storage, and the output is the played audio. Specifically, the application's audio player reads the audio data and sends it to the audio output device (speakers or earphones) for playback.

[1036] This enables users to perform secure voice authentication for multi-stage authentication, and enables safe and efficient transactions of voice data with identifiers.

[1037] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1038] The present invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and then stores and trades it in a database, and further combines it with an emotion engine to increase the naturalness and diversity of the voice data. The system of the present invention solves these problems by having the user, server, and terminal work together.

[1039] User processing

[1040] Audio Recording and Transmission

[1041] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[1042] Buying and selling

[1043] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[1044] Server-side processing

[1045] Speech Analysis and Generation

[1046] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[1047] Combining Emotion Engines

[1048] In this invention, an emotion engine is built into the server, which can recognize emotions from the user's voice data. The emotion engine acquires the user's emotion data and reflects it in the generated voice data. For example, if the user is speaking with a cheerful tone, that emotion can be reflected in the tone and pitch of the voice data.

[1049] Assigning identifiers

[1050] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[1051] Marketplace operations

[1052] The server publishes voice data that users have listed on the marketplace. When a user purchases a voice data item, the voice data is downloaded to the buyer's device. The server also manages transactions, notifies buyers, and processes sellers' compensation. It can also automatically categorize voice data using emotion data recognized by the emotion engine.

[1053] Terminal side processing

[1054] Download and play audio

[1055] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[1056] Data Management and Backup

[1057] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[1058] Specific examples

[1059] As an example, consider a scenario in which a user is recording narration for a presentation. The user uses a dedicated application to record audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. An emotion engine is also used to recognize the user's emotions and reflect them in the generated audio data. This data is then put up for sale on a marketplace where other users can purchase it. The audio data downloaded by the purchaser is then played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[1060] By implementing these functions, the present invention solves the problems of users' voices and provides new value to voice data. By combining it with an emotion engine, it becomes possible to generate even more natural and emotionally rich voice data.

[1061] The processing flow will be explained below.

[1062] User processing steps

[1063] Step 1:

[1064] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[1065] Step 2:

[1066] The user presses the "Start Recording" button to begin recording the audio.

[1067] Step 3:

[1068] When the user has finished speaking, he or she presses the "End Recording" button to stop recording.

[1069] Step 4:

[1070] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[1071] Step 5:

[1072] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[1073] ---

[1074] Server-side processing steps

[1075] Step 1:

[1076] The server receives the voice data sent by the user and temporarily stores it.

[1077] Step 2:

[1078] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[1079] Step 3:

[1080] The server uses generative AI technology to generate new voice data based on the text data, and is equipped with an emotion engine that recognizes the user's emotions from the voice data.

[1081] Step 4:

[1082] The emotion engine analyzes the recognized emotion data and reflects it in the generated voice data. For example, if the user is speaking with a happy expression, that emotion will be reflected in the tone and pitch of the voice data.

[1083] Step 5:

[1084] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[1085] Step 6:

[1086] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[1087] Step 7:

[1088] The server publishes the voice data that users have put up for sale on the marketplace and manages the transactions of the voice data. When a transaction is completed, the server provides the voice data to the buyer and pays a fee to the seller. The server also automatically categorizes the voice data using emotion data recognized by the emotion engine.

[1089] ---

[1090] Terminal processing steps

[1091] Step 1:

[1092] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[1093] Step 2:

[1094] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[1095] Step 3:

[1096] When a user inputs text data in real time, the device transmits the data to the server and instantly plays back the generated voice data, with the emotion engine reflecting appropriate emotions in the voice data based on the text data entered by the user.

[1097] Step 4:

[1098] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[1099] In this way, a system is realized in which the user, server, and terminal work together to effectively manage and utilize the user's voice data. In addition, the introduction of an emotion engine greatly improves the naturalness and emotional impact of the voice data.

[1100] Example 2

[1101] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1102] The present invention aims to generate more natural and diverse voice data by reflecting emotions when analyzing and generating voice data. Another objective of the present invention is to improve the reliability and convenience of voice data by realizing secure transaction management of voice data with assigned identifiers and cloud backup of voice data.

[1103] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing emotions from user voice data and reflecting the emotions in generated voice data, a means for saving multiple voice data input by the user in local storage and backing them up to the cloud as needed, and a means for managing transactions of voice data in the marketplace. This makes it possible to generate more natural and emotional voice data that reflects emotions, and also improve the reliability and safety of the voice data.

[1104] "User" refers to any individual or entity that records, transmits, buys, or sells audio data.

[1105] "Server" refers to a central management system for analyzing, generating, and managing voice data sent by users.

[1106] "Terminal" refers to the device used by a user (e.g., smartphone, tablet, computer, etc.).

[1107] "Audio Data" refers to digital files that contain recordings of a user's voice.

[1108] "Text data" refers to character information that has been converted from analyzed voice data.

[1109] "Identifier" refers to a unique identification number or NFT assigned to audio data.

[1110] "Marketplace" refers to an online platform for trading voice data.

[1111] An "emotion engine" refers to a system that recognizes a user's emotions from voice data and reflects them in the generated voice data.

[1112] "Local storage" refers to a storage area for storing data within a terminal.

[1113] "Cloud" refers to data storage provided on remote servers accessible over the Internet.

[1114] A "generative AI model" refers to an artificial intelligence algorithm for generating new voice data based on text data.

[1115] This invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and stores and trades it in a database. The main purpose is to increase the naturalness and diversity of voice data by combining it with an emotion engine. The system operates in cooperation with the user, server, and terminal.

[1116] User processing

[1117] Audio Recording and Transmission

[1118] The user starts the dedicated application and starts recording audio. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. This transmission process is carried out automatically via the Internet.

[1119] Buying and selling

[1120] Users can access the marketplace and list their own audio data or purchase audio data listed by other users. The marketplace provides an interface for setting the selling price. For example, a user can record a narration for a presentation and list it for 500 yen. The following is an example of a prompt:

[1121] "Hello everyone. Today I'd like to talk to you about our new project."

[1122] Server-side processing

[1123] Speech Analysis and Generation

[1124] The server receives the voice data sent by the user and temporarily stores it. It then uses Google Cloud's speech recognition API to analyze the voice data and convert it into text data. This text data is then input back into a speech generation system (e.g., a generative AI model) to generate new voice data. A generative AI model such as OpenAI's GPT-3 could be used.

[1125] Combining Emotion Engines

[1126] The server is equipped with an emotion engine that recognizes emotions from the user's voice data. It acquires emotion data using tools such as IBM Watson Tone Analyzer and reflects that emotion in the generated voice data. For example, if the user sounds like they're having fun, that emotion can be reflected in the tone and pitch of the voice data.

[1127] Assigning identifiers

[1128] The generated voice data is assigned an identifier, such as an NFT (Non-Fungible Token). The identifiers are managed using the Ethereum blockchain or similar to prove that the voice data is unique. After the identifier is assigned, the voice data is stored in a database.

[1129] Marketplace operations

[1130] The server publishes the voice data that users have put up for sale on the marketplace. When a user purchases voice data, the server downloads the data to the buyer's device, manages the transaction, notifies the buyer, and processes compensation for the seller. It also automatically sets the category of the voice data using data from the emotion engine.

[1131] Terminal side processing

[1132] Download and play audio

[1133] When a user purchases audio data from the marketplace, it is automatically downloaded to the device and can be played using the player function within the application. It is also possible to generate audio based on text data entered in real time and play it instantly.

[1134] Data Management and Backup

[1135] The user's device will save downloaded audio data to local storage as needed. In addition, important audio data will be backed up to the cloud periodically. Backups can be automated using the Google Drive API, for example.

[1136] Specific examples

[1137] For example, consider a scenario where a user is recording narration for a presentation. The user uses a dedicated app to record audio and send it to the server. The server then uses Google Cloud's speech recognition API to convert the audio data into text and generates new audio data using the OpenAI GPT-3 model. This audio data is assigned an NFT identifier using the Ethereum blockchain. An emotion engine is also used to recognize emotions and reflect them in tone and pitch. This data is then put up for sale on a marketplace where other users can purchase it. Buyers can play the downloaded audio data using the player function. Real-time audio generation is also possible, allowing for audio substitution even if the user is unable to speak.

[1138] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1139] Step 1:

[1140] The user launches a dedicated application. They press the "Start Recording" button in the application and then speak into the microphone to record audio data. When recording is complete, they press the "Stop Recording" button. The input is the user's voice, and the output is an audio file (e.g., WAV format). Specifically, the user taps a button on the app screen on their smartphone.

[1141] Step 2:

[1142] The user presses the "Send" button to send the recorded audio data to the server. The data is automatically uploaded to the server via the Internet. The input is the recorded audio file, and the output is the audio file saved on the server. During the sending process, a progress bar is displayed on the screen.

[1143] Step 3:

[1144] The server temporarily stores the received voice data. The stored voice file is used as input data for analysis processing. Specifically, the received data is stored in a temporary folder.

[1145] Step 4:

[1146] The server converts the voice data into text data using Google Cloud's speech recognition API. Here, the voice file is input and the converted text data is output. A specific API call is made to convert the voice data into text information.

[1147] Step 5:

[1148] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate new voice data from text data. The input is text data, and the output is a new voice file. In this case, the voice data is generated by giving a prompt to the AI ​​model. For example, the following prompt sentence is used:

[1149] "Hello everyone. Today I'd like to talk to you about our new project."

[1150] Step 6:

[1151] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the original audio data. This module takes an audio file as input and outputs emotional data, which is reflected in the tone and pitch of the voice.

[1152] Step 7:

[1153] The generated audio data is assigned an identifier (e.g., NFT). The server generates the identifier using blockchain technology (e.g., Ethereum) and associates it with the audio data. The input is an audio file, and the output is an audio file with an identifier.

[1154] Step 8:

[1155] The server stores the voice data with the identifier in a database. The voice files with the identifier are stored in a database, and the database manages the voice data and maintains a transaction history.

[1156] Step 9:

[1157] The server publishes listing information for audio data sold by users on the marketplace. The input is an audio file with an identifier and a selling price, and the output is the product information published on the marketplace. The server actually updates the interface to display seller information and a preview of the audio.

[1158] Step 10:

[1159] When a user purchases audio data from the marketplace, the audio data is downloaded to the user's device. The input is a purchase instruction and an audio file with an identifier, and the output is the audio file saved on the user's device. Specifically, the download begins automatically once the purchase process is complete.

[1160] Step 11:

[1161] The device plays the downloaded audio data using the application's player function. The input is the downloaded audio file, and the output is the played audio. The audio starts playing when the user presses the play button in the app.

[1162] Step 12:

[1163] The user's device saves downloaded audio data to local storage and periodically backs it up to the cloud. The input is the audio file in local storage, and the output is the audio file stored in the cloud. Specifically, if the automatic backup setting is enabled, the data will be uploaded to cloud storage at the specified time.

[1164] (Application example 2)

[1165] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1166] Conventional voice generation services analyze user-recorded voice data and lack naturalness and diversity when using the generated voice data. Furthermore, there are no systems that can effectively respond to situations requiring real-time voice responses that reflect the user's emotions. This leads to insufficient customer service in virtual stores and a poor user experience.

[1167] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user voice data and acquiring text data and emotion data, means for generating natural voice data from the text data and emotion data, and means for assigning a unique identifier to the generated voice data and storing it in a database. This makes it possible to generate natural and diverse voice data that reflects emotions, and realizes real-time user responses in virtual stores, etc.

[1168] "Means for recording the user's voice" refers to a device or software for recording the user's spoken voice as digital data.

[1169] "Means for transmitting recorded voice data to a server" refers to a device or software that has the function of transferring recorded voice data to a server via a network.

[1170] "Means for analyzing voice data received by the server and converting it into text data" refers to technology for processing voice data received by the server and converting it into text in natural language.

[1171] "Means for obtaining emotional data from analyzed text data" refers to technology that uses analyzed text data to recognize and determine the emotional state of the speaker.

[1172] "Means for generating voice data from text data and emotional data" refers to technology that creates natural, emotionally rich voice data based on text data and emotional data.

[1173] "Means for assigning an identifier to generated voice data" refers to a technology for adding unique identification information to generated voice data to prove the authenticity and uniqueness of the data.

[1174] "Means for storing voice data with identifiers in a database" refers to a technology for securely storing voice data with unique identification information added in a database.

[1175] "Means for listing voice data with identifiers on a marketplace" refers to technology that makes voice data with identifiers attached publicly available on an online platform and makes it available for sale or trading.

[1176] "Means for managing transactions of voice data in a marketplace" refers to technology that oversees the buying, selling, and exchange of voice data on a marketplace and ensures that transactions are carried out smoothly.

[1177] "Means for downloading audio data purchased by a user to a terminal" refers to technology for saving audio data purchased by a user on a marketplace to the user's device.

[1178] "Means for playing downloaded audio data on a terminal" refers to software or hardware for playing audio data downloaded to a terminal.

[1179] "Means for generating and instantly playing back audio data in real time" refers to technology for instantly generating audio data based on audio or text input by a user and playing it back in real time.

[1180] The system for implementing this invention comprises a voice recording means, a server communication means, a voice analysis and generation means, an identifier assignment means, a database storage means, a marketplace management means, a download means, a playback means, a real-time voice generation means, and an emotion analysis means.

[1181] User processing

[1182] The user launches a dedicated application using a smartphone or head-mounted display (HMD). The application provides "Start Recording" and "Stop Recording" buttons. The user operates these buttons to record audio, and then presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[1183] Server-side processing

[1184] The server first temporarily stores the received voice data. It then converts the voice data into text data using the Google Cloud Speech-to-Text API. At the same time, it analyzes emotion data from the text data using the Emotion API. It then uses the Google Cloud Text-to-Speech API to generate natural-sounding voice data based on the text data and emotion data. The generated voice data is assigned an identifier using an NFT (non-fungible token) and stored in a database.

[1185] Marketplace operations

[1186] The server also includes a function for listing voice data with identifiers on the marketplace. Users can access the marketplace and list or purchase voice data. The server is also responsible for setting prices and managing transactions for voice data on the marketplace. Users who purchase voice data can download the voice data to their own devices via the cloud service.

[1187] Terminal side processing

[1188] The downloaded voice data can be played back on the user's device within a dedicated application. It is also possible to generate voice based on text data entered in real time and play it back instantly. By providing natural, rich voice responses that reflect emotion analysis, customer service in virtual stores is enhanced.

[1189] Examples of concrete examples and prompts

[1190] For example, consider the case where a user inquires about trying on clothes in a virtual store. When the user says, "Do you prefer this outfit in black or red?", the voice is recorded and sent to the server. The server analyzes the voice and obtains text data and emotion data. A voice response generated based on this data, "Black clothes are chic and lovely," is sent to the user's device and played back.

[1191] Example prompt for generative AI model:

[1192] "Write a Python program that converts the user's voice data into text, analyzes the emotion data, and then generates the speech again. Use the Google Cloud Speech-to-Text and Google Cloud Text-to-Speech APIs, and the Emotion API for emotion analysis."

[1193] This invention makes it possible to generate natural and diverse voice data that reflects emotions, thereby enhancing customer service in virtual stores, etc. This is expected to improve the user experience.

[1194] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1195] Step 1:

[1196] The user starts the dedicated application and clicks the "Start Recording" button to record audio.

[1197] Input: User utterance

[1198] Output: Recorded audio data

[1199] Specific operation: Records the user's voice using the microphone built into the smartphone or head-mounted display (HMD) and saves the audio file in local storage.

[1200] Step 2:

[1201] The user clicks the "End Recording" button and then presses the "Send" button to send the recorded voice data to the server.

[1202] Input: Recorded audio data

[1203] Output: Audio data sent to the server

[1204] Specific behavior: A process is executed to upload an audio file to a server via the Internet.

[1205] Step 3:

[1206] The server temporarily stores the received audio data.

[1207] Input: Audio data sent to the server

[1208] Output: Temporarily saved audio data

[1209] What it does: Saves the audio file to the server's local storage or temporary cache area.

[1210] Step 4:

[1211] The server uses the Google Cloud Speech-to-Text API to analyze the audio data and convert it into text data.

[1212] Input: Temporarily stored audio data

[1213] Output: Parsed text data

[1214] Specific operation: Sends an audio file to the API and retrieves the resulting string data as text data.

[1215] Step 5:

[1216] The server uses the Emotion API to obtain emotion data from text data.

[1217] Input: Parsed text data

[1218] Output: Emotion data

[1219] What it does: Sends text data to the Emotion API and retrieves data indicating the emotional state based on that text.

[1220] Step 6:

[1221] The server uses the Google Cloud Text-to-Speech API to generate voice data based on the text data and emotion data.

[1222] Input: Text data and emotion data

[1223] Output: Generated audio data

[1224] What it does: Sends text and emotion data to the API and retrieves the generated audio file.

[1225] Step 7:

[1226] The server assigns an NFT identifier to the generated audio data and stores it in a database.

[1227] Input: Generated audio data

[1228] Output: Audio data with identifiers

[1229] What it does: Adds a unique identifier to the audio file and stores it in a database.

[1230] Step 8:

[1231] The server puts the voice data with the identifier on the marketplace.

[1232] Input: Audio data with identifiers

[1233] Output: Audio data listed on the marketplace

[1234] What it does: Publishes audio files through the marketplace's management system, making them accessible to other users.

[1235] Step 9:

[1236] When a user purchases audio data from the marketplace, the data is downloaded to the device.

[1237] Input: Audio data selected from the marketplace

[1238] Output: Audio data downloaded to the device

[1239] Specific operation: Transfer data from cloud storage to the user's device and save it in local storage.

[1240] Step 10:

[1241] The terminal plays the downloaded audio data.

[1242] Input: Audio data downloaded to the device

[1243] Output: The audio data to be played

[1244] Specific Behavior: Plays an audio file using the audio player functionality within the application.

[1245] Step 11:

[1246] The device generates audio based on the text data entered in real time and plays it back instantly.

[1247] Input: Text data entered by the user

[1248] Output: Audio data generated and played in real time

[1249] Specific operation: Sends text data to the server, receives real-time generated audio data, and plays it back immediately.

[1250] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1251] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1252] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1253] [Fourth embodiment]

[1254] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1255] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1256] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1257] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1258] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1259] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1260] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1261] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1262] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1263] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1264] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1265] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1266] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1267] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data. The system of the present invention solves the problem by having the user, server, and terminal work together as shown below.

[1268] User processing

[1269] Audio Recording and Transmission

[1270] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[1271] Buying and selling

[1272] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[1273] Server-side processing

[1274] Speech Analysis and Generation

[1275] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[1276] Assigning identifiers

[1277] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[1278] Marketplace operations

[1279] The server publishes the audio data that users have listed on the marketplace. When a user purchases the data, the audio data is downloaded to the buyer's device. The server also manages transactions, notifies the buyer, and processes the seller's compensation.

[1280] Terminal side processing

[1281] Download and play audio

[1282] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[1283] Data Management and Backup

[1284] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[1285] Specific examples

[1286] As an example, consider a scenario where a user wants to record a narration for a presentation. The user uses a dedicated application to record the audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. Users can access a marketplace where they can sell their own audio data or purchase audio data from other users. The audio data is downloaded and played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[1287] By implementing these functions, the present invention solves the problems associated with users' voices and provides new value to voice data.

[1288] The processing flow will be explained below.

[1289] User processing steps

[1290] Step 1:

[1291] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[1292] Step 2:

[1293] The user presses the "Start Recording" button to start recording the audio, and when they have finished speaking, they press the "Stop Recording" button to stop recording.

[1294] Step 3:

[1295] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[1296] Step 4:

[1297] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[1298] Server-side processing steps

[1299] Step 1:

[1300] The server receives the voice data sent by the user and temporarily stores it.

[1301] Step 2:

[1302] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[1303] Step 3:

[1304] The server generates new voice data based on the text data using generative AI technology.

[1305] Step 4:

[1306] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[1307] Step 5:

[1308] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[1309] Step 6:

[1310] The server publishes the audio data that users have put up for sale on the marketplace and manages transactions of the audio data. When a transaction is completed, the audio data is provided to the buyer and a fee is paid to the seller.

[1311] Terminal processing steps

[1312] Step 1:

[1313] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[1314] Step 2:

[1315] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[1316] Step 3:

[1317] When a user inputs text data in real time, the terminal transmits this data to the server and instantly plays back the generated voice data.

[1318] Step 4:

[1319] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[1320] In this way, a system is realized in which the user, server, and terminal work in cooperation with each other to effectively manage and utilize the user's voice data.

[1321] Example 1

[1322] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1323] Counterfeiting and unauthorized use of voice data have become a problem in the current voice data trading market. Conventional technologies lack a method to guarantee the uniqueness and authenticity of data when users record and trade their own voice data. This reduces the reliability of voice data and poses a challenge to ensuring transparency in transactions. Furthermore, the user experience is insufficient in the generation and management of voice data itself, as well as in post-purchase usage.

[1324] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1325] In this invention, the server includes means for recording the user's voice, means for transmitting the recorded voice data to the data server, means for analyzing the voice data received by the data server and converting it into text data, means for generating voice data from the text data, means for assigning an identifier to the generated voice data, means for storing the identified voice data in a database, means for listing the identified voice data on an online market, means for managing transactions of voice data on the online market, means for downloading voice data purchased by the user to an end-user terminal, and means for playing the voice data. This ensures the uniqueness and authenticity of the voice data and enables transparent transactions. Furthermore, users can efficiently generate, manage, trade, and use voice data.

[1326] "User" refers to any person or entity that uses the System to record voice and make transactions or purchases.

[1327] "Audio data" refers to data that represents audio information recorded by a user in digital form.

[1328] "Data Server" refers to a computer system that receives, analyzes, and stores voice data sent by users and manages transactions of voice data.

[1329] "Analysis" refers to the process of converting received voice data into text data.

[1330] "Text data" refers to character information obtained by analyzing voice data.

[1331] "Speech generation" refers to the process of generating new voice data based on the text data obtained through analysis.

[1332] "Identifier" refers to a digital tag or code that is attached to generated audio data to ensure the uniqueness and authenticity of the audio data.

[1333] "Database" refers to a storage device or storage system for storing identified audio data.

[1334] "Electronic Marketplace" refers to an online platform for trading identified voice data.

[1335] "End-user device" refers to a user's device (e.g., smartphone, personal computer) used to record, transmit, download, or play audio data.

[1336] "Playback" refers to the process of outputting audio data downloaded to an end-user terminal as an acoustic signal.

[1337] MODE FOR CARRYING OUT THE INVENTION

[1338] The present invention relates to a voice generation service that records a user's voice, transmits it to a server, analyzes and regenerates the voice data, assigns an identifier, and stores and trades the data.

[1339] System Overview

[1340] The system uses the following hardware and software:

[1341] The device used by the user (e.g. smartphone, PC)

[1342] Dedicated application for recording and playback

[1343] Data server (receives, analyzes, and generates voice data)

[1344] Speech recognition technology for speech analysis (e.g., Google Cloud Speech-to-Text API)

[1345] AI models for speech generation (e.g., OpenAI GPT-3)

[1346] Database (storing voice data and identifiers)

[1347] Electronic market (voice data trading platform)

[1348] User processing

[1349] The user first launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, and the user presses the "Start Recording" button to begin recording. When recording is finished, the user presses the "Stop Recording" button, and then the "Send" button to upload the audio data to the server. At this time, the data is automatically sent via the Internet.

[1350] Server-side processing

[1351] The server temporarily stores the received voice data and performs voice analysis. The Google Cloud Speech-to-Text API is used to convert the voice data into text. The text data is then input into an AI model such as OpenAI GPT-3 to generate new voice data. This generated voice data is given an identifier such as an NFT to prevent counterfeiting or fraudulent use, and is stored in a database.

[1352] Terminal side processing

[1353] When a user purchases audio data from an online marketplace, the data is downloaded to the user's device via the Internet. The downloaded audio data is played using the player function within a dedicated application. Real-time audio generation is also possible, and audio data is instantly generated and played based on text data entered by the user.

[1354] Specific examples

[1355] As an example, consider a scenario in which a user wants to record narration for a presentation. The user launches a dedicated application and presses the "Start Recording" button to record the narration. When the recording is complete, the user presses the "Stop Recording" and "Send" buttons to send the audio data to the server. The server analyzes the audio data, converts it into text data, and then generates new audio data. This audio data is assigned an identifier and stored in a database. The user can then list the audio data on an electronic marketplace.

[1356] Prompt Sentence Examples

[1357] Please provide detailed instructions on how users can use the dedicated application to record audio and send the audio data to the server. Please also include details of the interface and the names of the operation buttons.

[1358] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1359] Step 1:

[1360] The user launches a dedicated application. The application screen displays a "Start Recording" button and a "Stop Recording" button. When the user presses the "Start Recording" button, the device's microphone is enabled and audio recording begins. The input is the user's voice, and the output is the recorded audio data.

[1361] Step 2:

[1362] When the user presses the "Stop Recording" button to end recording, the audio recording stops. The device temporarily stores this recording data in local storage. The input is an instruction to stop audio recording, and the output is the temporarily stored audio data.

[1363] Step 3:

[1364] The user presses the "Send" button to send the recorded data to the server. The device then uploads the audio data to the server via the Internet. The input is the temporarily saved audio data, and the output is the audio data sent to the server.

[1365] Step 4:

[1366] The server stores the received audio data in a temporary storage area. At this point, it performs an integrity check on the data to ensure it is not corrupted. The input is the transmitted audio data, and the output is the audio data stored in the temporary storage area.

[1367] Step 5:

[1368] The server starts the speech analysis. Specifically, it converts the speech data into text data using the Google Cloud Speech-to-Text API. The input is the speech data in the temporary storage area, and the output is the analyzed text data.

[1369] Step 6:

[1370] The server inputs the text data into an AI speech generation model (e.g., OpenAI GPT-3) to generate new speech data. The input is the analyzed text data, and the output is the generated speech data.

[1371] Step 7:

[1372] The server assigns an identifier such as an NFT to the generated audio data. This procedure guarantees the uniqueness and authenticity of the audio data. The input is the generated audio data, and the output is audio data with an identifier.

[1373] Step 8:

[1374] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database.

[1375] Step 9:

[1376] Users access the marketplace to sell their own voice data or purchase other users' voice data. The marketplace provides an interface for managing transactions and setting prices. The input is a marketplace access instruction, and the output is transaction information.

[1377] Step 10:

[1378] When a user purchases audio data, the server downloads the audio data to the end-user terminal. The input is a purchase instruction, and the output is the audio data downloaded to the end-user terminal.

[1379] Step 11:

[1380] The device plays the downloaded audio data within a dedicated application. Pressing the play button starts audio playback. The input is the downloaded audio data, and the output is the played audio.

[1381] (Application example 1)

[1382] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1383] Conventional voice authentication systems have had problems with the accuracy and security of voice data authentication, and are not resistant to fraudulent use or attacks. Furthermore, there is a lack of transparency in the management and trading of voice authentication data, making it difficult to ensure trust between users. Furthermore, in addition to voice authentication, marketplace functions to increase the added value of voice data are lacking.

[1384] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1385] In this invention, the server includes a means for recording the user's voice, a means for transmitting the recorded voice data to the server, and a means for analyzing the received voice data and converting it into text data. This improves the accuracy of voice data authentication, prevents fraudulent use, and enables safe and efficient trading of voice data through the marketplace.

[1386] A "user" is an entity that uses the system and records, transmits, and plays back audio data.

[1387] "Voice data" is digital data that is a recording of a user's speech.

[1388] A "server" is a computer system that receives, analyzes, generates, identifies, stores, and manages voice data via a network.

[1389] "Analysis" is the process of processing received voice data and converting it into text data.

[1390] "Text data" is character information converted from audio data through analysis.

[1391] An "identifier" is unique information that is assigned to generated voice data to prevent counterfeiting or unauthorized use of the voice data.

[1392] "Marketplace" means an online platform where audio data is traded.

[1393] "Authentication" is the process of verifying a user's authenticity using recorded voice data.

[1394] "Matching" is the process of comparing generated and recorded audio data to confirm a match.

[1395] "Downloading" is the act of transferring audio data from a server to a user's terminal.

[1396] "Playback" refers to the act of outputting downloaded audio data as sound again on the terminal.

[1397] A system for implementing this invention requires the following components to record, transmit, analyze, generate, identify, store, and transact user voice data:

[1398] 1. User-side processing

[1399] Hardware and software:

[1400] Smartphone: Microphone, Storage

[1401] Dedicated applications: voice recording, transmission, player, etc.

[1402] First, users launch the dedicated application installed on their smartphone and tap the recording start button to record their own voice. When recording is finished, they press the send button to send the recorded voice data to the server. The voice data is automatically uploaded to the server via the Internet. Users can also access the marketplace to list their own voice data or purchase other voice data.

[1403] 2. Server-side processing

[1404] Hardware and software:

[1405] Cloud server: Database, speech analysis system, speech generation system, identifier assignment system, etc.

[1406] Google Cloud Speech-to-Text API: An analysis system for converting voice data into text data.

[1407] AI model: A system for generating new voice data from text data

[1408] The server receives the voice data sent by the user and temporarily stores it. The voice analysis system then converts the received voice data into text data. The text data is then input back into the voice generation system, and new voice data is generated using AI technology. A unique identifier is assigned to the generated voice data using an identifier assignment system. The voice data with the identifier assigned is stored in a database and made public on the marketplace. The marketplace manages downloads and transactions of voice data purchased by users.

[1409] 3. Processing on the user terminal side

[1410] Hardware and software:

[1411] Smartphone: Storage, player functions

[1412] Cloud service: Regular backup function

[1413] When a user purchases audio data from the marketplace, the audio data is downloaded to their smartphone. The downloaded audio data is played using the player function within the application. The audio data is also saved in local storage as needed and periodically backed up to the cloud.

[1414] Specific examples

[1415] For example, when a user wants to unlock their smartphone, they record their voice using a dedicated application, and after recording is complete, send it to the server. The server analyzes the voice and compares it with the newly generated voice data for authentication. At this time, the voice data with an identifier is stored in a database, preventing unauthorized use. If authentication is successful, the smartphone is unlocked.

[1416] Prompt Sentence Examples

[1417] Describe a scenario in which a user attempts to unlock a smartphone using a specific voice. Explain in detail the process from recording the user's voice and sending it to the server, to analyzing the voice on the server side, assigning an identifier, and finally multi-factor authentication.

[1418] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1419] Step 1:

[1420] The user records audio

[1421] The user launches a dedicated application on their smartphone and taps the "Start Recording" button to record their own voice. When the recording is complete, they press the "Stop Recording" button. In this step, the voice input from the user's microphone is saved as digital voice data. Specifically, the input data is the user's voice, and the output is an audio file saved on the smartphone.

[1422] Step 2:

[1423] Sending audio data from the device to the server

[1424] The user taps the "Send" button to send the recorded audio data to the server. The smartphone application uploads the audio data to the server as an HTTP POST request. The input is the recorded audio data, and the output is the audio data saved on the server. Specifically, the application saves the audio file as a temporary file and sends it to the server via the Internet.

[1425] Step 3:

[1426] The server analyzes the voice data and converts it into text data.

[1427] The server analyzes the received voice data and converts it into text data using a voice analysis system (e.g., Google Cloud Speech-to-Text API). The input is voice data, and the output is the converted text data. Specifically, the server sends the voice data to the API and receives the converted text.

[1428] Step 4:

[1429] The server generates new voice data from the text data.

[1430] The server inputs the text data into a speech generation system to generate new speech data. A generative AI model is used here. The input is text data, and the output is newly generated speech data. Specifically, the AI ​​speech generation model analyzes the text data and generates a speech file.

[1431] Step 5:

[1432] The server assigns an identifier to the generated audio data.

[1433] The server uses an identification system to assign a unique identifier (e.g., NFT) to the generated audio data. The input is the generated audio data, and the output is audio data with an identifier. Specifically, the audio data is processed using a hash function or similar to generate the identifier.

[1434] Step 6:

[1435] Save the identified audio data in a database

[1436] The server stores the voice data with the identifier in a database. The input is the voice data with the identifier, and the output is the voice data stored in the database. Specifically, the voice data is stored in a fixed format using a database management system.

[1437] Step 7:

[1438] Publishing audio data with identifiers to the marketplace

[1439] The server publishes the identified voice data to the marketplace, allowing users to purchase and trade. The input is the voice data stored in the database, and the output is the voice data posted on the marketplace. Specifically, the web server displays the voice data through the marketplace interface.

[1440] Step 8:

[1441] The user purchases the audio data and downloads it to their device.

[1442] Users purchase audio data from the marketplace and download it to their smartphone. The input is the audio data on the marketplace, and the output is the audio data stored on the smartphone. Specifically, the application downloads the audio data using an HTTP request and saves it in local storage.

[1443] Step 9:

[1444] Play audio data on the device

[1445] The user plays the downloaded audio data using the player function of the dedicated application. The input is the audio data stored in local storage, and the output is the played audio. Specifically, the application's audio player reads the audio data and sends it to the audio output device (speakers or earphones) for playback.

[1446] This enables users to perform secure voice authentication for multi-stage authentication, and enables safe and efficient transactions of voice data with identifiers.

[1447] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1448] The present invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and then stores and trades it in a database, and further combines it with an emotion engine to increase the naturalness and diversity of the voice data. The system of the present invention solves these problems by having the user, server, and terminal work together.

[1449] User processing

[1450] Audio Recording and Transmission

[1451] First, the user launches the dedicated application. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[1452] Buying and selling

[1453] Users can access the marketplace to sell their own audio data or purchase other audio data, and the marketplace provides an interface for users to set selling prices.

[1454] Server-side processing

[1455] Speech Analysis and Generation

[1456] The server receives the voice data sent by the user. The received voice data is first temporarily stored and then analyzed. During the voice analysis stage, the voice data is converted into text data. After analysis, the text data is again input into the voice generation system, where new voice data is generated using AI technology.

[1457] Combining Emotion Engines

[1458] In this invention, an emotion engine is built into the server, which can recognize emotions from the user's voice data. The emotion engine acquires the user's emotion data and reflects it in the generated voice data. For example, if the user is speaking with a cheerful tone, that emotion can be reflected in the tone and pitch of the voice data.

[1459] Assigning identifiers

[1460] The generated audio data is given an identifier such as an NFT to prevent counterfeiting or unauthorized use. This identification process proves that the audio data is unique. After the identifier is assigned, the audio data is stored in a database.

[1461] Marketplace operations

[1462] The server publishes voice data that users have listed on the marketplace. When a user purchases a voice data item, the voice data is downloaded to the buyer's device. The server also manages transactions, notifies buyers, and processes sellers' compensation. It can also automatically categorize voice data using emotion data recognized by the emotion engine.

[1463] Terminal side processing

[1464] Download and play audio

[1465] When a user purchases audio data from the marketplace, the data is downloaded to the user's device. The downloaded audio data can be played using the application's in-app player. The application can also generate audio based on real-time text input and play it instantly.

[1466] Data Management and Backup

[1467] The user's device will save the downloaded audio data to local storage as needed for playback, and important audio data will be backed up to the cloud on a regular basis.

[1468] Specific examples

[1469] As an example, consider a scenario in which a user is recording narration for a presentation. The user uses a dedicated application to record audio and then sends it to a server. The server analyzes the audio data, converts it into text, and generates new audio data. This audio data is assigned an identifier and stored in a database. An emotion engine is also used to recognize the user's emotions and reflect them in the generated audio data. This data is then put up for sale on a marketplace where other users can purchase it. The audio data downloaded by the purchaser is then played back when needed. Real-time audio generation is also possible, allowing for substitution if the user is unable to speak.

[1470] By implementing these functions, the present invention solves the problems of users' voices and provides new value to voice data. By combining it with an emotion engine, it becomes possible to generate even more natural and emotionally rich voice data.

[1471] The processing flow will be explained below.

[1472] User processing steps

[1473] Step 1:

[1474] The user launches a dedicated application, which displays a "Start Recording" button and a "Stop Recording" button.

[1475] Step 2:

[1476] The user presses the "Start Recording" button to begin recording the audio.

[1477] Step 3:

[1478] When the user has finished speaking, he or she presses the "End Recording" button to stop recording.

[1479] Step 4:

[1480] Once the recording is complete, the user presses the "Send" button to send the recorded audio data to the server, where it is uploaded via the Internet.

[1481] Step 5:

[1482] Users access the marketplace to list their own voice data or search and select other voice data to purchase.

[1483] ---

[1484] Server-side processing steps

[1485] Step 1:

[1486] The server receives the voice data sent by the user and temporarily stores it.

[1487] Step 2:

[1488] The server analyzes the received voice data and converts it into text data. This analysis process uses acoustic processing technology.

[1489] Step 3:

[1490] The server uses generative AI technology to generate new voice data based on the text data, and is equipped with an emotion engine that recognizes the user's emotions from the voice data.

[1491] Step 4:

[1492] The emotion engine analyzes the recognized emotion data and reflects it in the generated voice data. For example, if the user is speaking with a happy expression, that emotion will be reflected in the tone and pitch of the voice data.

[1493] Step 5:

[1494] An identifier is assigned to the generated voice data, which guarantees the uniqueness of the voice data.

[1495] Step 6:

[1496] The voice data with the identifier is stored in a database, and the stored voice data is reflected in the marketplace listing.

[1497] Step 7:

[1498] The server publishes the voice data that users have put up for sale on the marketplace and manages the transactions of the voice data. When a transaction is completed, the server provides the voice data to the buyer and pays a fee to the seller. The server also automatically categorizes the voice data using emotion data recognized by the emotion engine.

[1499] ---

[1500] Terminal processing steps

[1501] Step 1:

[1502] The user's device downloads the audio data purchased from the marketplace from the server, and the data is stored in the user's local storage.

[1503] Step 2:

[1504] To play the audio data downloaded by the user, the application invokes the playback function. When the user presses the play button, the audio data is played through the speaker.

[1505] Step 3:

[1506] When a user inputs text data in real time, the device transmits the data to the server and instantly plays back the generated voice data, with the emotion engine reflecting appropriate emotions in the voice data based on the text data entered by the user.

[1507] Step 4:

[1508] The user's device stores downloaded audio data locally as needed and periodically backs up important data to the cloud.

[1509] In this way, a system is realized in which the user, server, and terminal work together to effectively manage and utilize the user's voice data. In addition, the introduction of an emotion engine greatly improves the naturalness and emotional impact of the voice data.

[1510] Example 2

[1511] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1512] The present invention aims to generate more natural and diverse voice data by reflecting emotions when analyzing and generating voice data. Another objective of the present invention is to improve the reliability and convenience of voice data by realizing secure transaction management of voice data with assigned identifiers and cloud backup of voice data.

[1513] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for recognizing emotions from user voice data and reflecting the emotions in generated voice data, a means for saving multiple voice data input by the user in local storage and backing them up to the cloud as needed, and a means for managing transactions of voice data in the marketplace. This makes it possible to generate more natural and emotional voice data that reflects emotions, and also improve the reliability and safety of the voice data.

[1514] "User" refers to any individual or entity that records, transmits, buys, or sells audio data.

[1515] "Server" refers to a central management system for analyzing, generating, and managing voice data sent by users.

[1516] "Terminal" refers to the device used by a user (e.g., smartphone, tablet, computer, etc.).

[1517] "Audio Data" refers to digital files that contain recordings of a user's voice.

[1518] "Text data" refers to character information that has been converted from analyzed voice data.

[1519] "Identifier" refers to a unique identification number or NFT assigned to audio data.

[1520] "Marketplace" refers to an online platform for trading voice data.

[1521] An "emotion engine" refers to a system that recognizes a user's emotions from voice data and reflects them in the generated voice data.

[1522] "Local storage" refers to a storage area for storing data within a terminal.

[1523] "Cloud" refers to data storage provided on remote servers accessible over the Internet.

[1524] A "generative AI model" refers to an artificial intelligence algorithm for generating new voice data based on text data.

[1525] This invention relates to a voice generation service that records a user's voice, sends it to a server for analysis and regeneration, assigns an identifier, and stores and trades it in a database. The main purpose is to increase the naturalness and diversity of voice data by combining it with an emotion engine. The system operates in cooperation with the user, server, and terminal.

[1526] User processing

[1527] Audio Recording and Transmission

[1528] The user starts the dedicated application and starts recording audio. The application has "Start Recording" and "Stop Recording" buttons, which the user operates to record audio. When recording is finished, the user presses the "Send" button to send the recorded audio data to the server. This transmission process is carried out automatically via the Internet.

[1529] Buying and selling

[1530] Users can access the marketplace and list their own audio data or purchase audio data listed by other users. The marketplace provides an interface for setting the selling price. For example, a user can record a narration for a presentation and list it for 500 yen. The following is an example of a prompt:

[1531] "Hello everyone. Today I'd like to talk to you about our new project."

[1532] Server-side processing

[1533] Speech Analysis and Generation

[1534] The server receives the voice data sent by the user and temporarily stores it. It then uses Google Cloud's speech recognition API to analyze the voice data and convert it into text data. This text data is then input back into a speech generation system (e.g., a generative AI model) to generate new voice data. A generative AI model such as OpenAI's GPT-3 could be used.

[1535] Combining Emotion Engines

[1536] The server is equipped with an emotion engine that recognizes emotions from the user's voice data. It acquires emotion data using tools such as IBM Watson Tone Analyzer and reflects that emotion in the generated voice data. For example, if the user sounds like they're having fun, that emotion can be reflected in the tone and pitch of the voice data.

[1537] Assigning identifiers

[1538] The generated voice data is assigned an identifier, such as an NFT (Non-Fungible Token). The identifiers are managed using the Ethereum blockchain or similar to prove that the voice data is unique. After the identifier is assigned, the voice data is stored in a database.

[1539] Marketplace operations

[1540] The server publishes the voice data that users have put up for sale on the marketplace. When a user purchases voice data, the server downloads the data to the buyer's device, manages the transaction, notifies the buyer, and processes compensation for the seller. It also automatically sets the category of the voice data using data from the emotion engine.

[1541] Terminal side processing

[1542] Download and play audio

[1543] When a user purchases audio data from the marketplace, it is automatically downloaded to the device and can be played using the player function within the application. It is also possible to generate audio based on text data entered in real time and play it instantly.

[1544] Data Management and Backup

[1545] The user's device will save downloaded audio data to local storage as needed. In addition, important audio data will be backed up to the cloud periodically. Backups can be automated using the Google Drive API, for example.

[1546] Specific examples

[1547] For example, consider a scenario where a user is recording narration for a presentation. The user uses a dedicated app to record audio and send it to the server. The server then uses Google Cloud's speech recognition API to convert the audio data into text and generates new audio data using the OpenAI GPT-3 model. This audio data is assigned an NFT identifier using the Ethereum blockchain. An emotion engine is also used to recognize emotions and reflect them in tone and pitch. This data is then put up for sale on a marketplace where other users can purchase it. Buyers can play the downloaded audio data using the player function. Real-time audio generation is also possible, allowing for audio substitution even if the user is unable to speak.

[1548] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1549] Step 1:

[1550] The user launches a dedicated application. They press the "Start Recording" button in the application and then speak into the microphone to record audio data. When recording is complete, they press the "Stop Recording" button. The input is the user's voice, and the output is an audio file (e.g., WAV format). Specifically, the user taps a button on the app screen on their smartphone.

[1551] Step 2:

[1552] The user presses the "Send" button to send the recorded audio data to the server. The data is automatically uploaded to the server via the Internet. The input is the recorded audio file, and the output is the audio file saved on the server. During the sending process, a progress bar is displayed on the screen.

[1553] Step 3:

[1554] The server temporarily stores the received voice data. The stored voice file is used as input data for analysis processing. Specifically, the received data is stored in a temporary folder.

[1555] Step 4:

[1556] The server converts the voice data into text data using Google Cloud's speech recognition API. Here, the voice file is input and the converted text data is output. A specific API call is made to convert the voice data into text information.

[1557] Step 5:

[1558] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate new voice data from text data. The input is text data, and the output is a new voice file. In this case, the voice data is generated by giving a prompt to the AI ​​model. For example, the following prompt sentence is used:

[1559] "Hello everyone. Today I'd like to talk to you about our new project."

[1560] Step 6:

[1561] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to recognize emotions from the original audio data. This module takes an audio file as input and outputs emotional data, which is reflected in the tone and pitch of the voice.

[1562] Step 7:

[1563] The generated audio data is assigned an identifier (e.g., NFT). The server generates the identifier using blockchain technology (e.g., Ethereum) and associates it with the audio data. The input is an audio file, and the output is an audio file with an identifier.

[1564] Step 8:

[1565] The server stores the voice data with the identifier in a database. The voice files with the identifier are stored in a database, and the database manages the voice data and maintains a transaction history.

[1566] Step 9:

[1567] The server publishes listing information for audio data sold by users on the marketplace. The input is an audio file with an identifier and a selling price, and the output is the product information published on the marketplace. The server actually updates the interface to display seller information and a preview of the audio.

[1568] Step 10:

[1569] When a user purchases audio data from the marketplace, the audio data is downloaded to the user's device. The input is a purchase instruction and an audio file with an identifier, and the output is the audio file saved on the user's device. Specifically, the download begins automatically once the purchase process is complete.

[1570] Step 11:

[1571] The device plays the downloaded audio data using the application's player function. The input is the downloaded audio file, and the output is the played audio. The audio starts playing when the user presses the play button in the app.

[1572] Step 12:

[1573] The user's device saves downloaded audio data to local storage and periodically backs it up to the cloud. The input is the audio file in local storage, and the output is the audio file stored in the cloud. Specifically, if the automatic backup setting is enabled, the data will be uploaded to cloud storage at the specified time.

[1574] (Application example 2)

[1575] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1576] Conventional voice generation services analyze user-recorded voice data and lack naturalness and diversity when using the generated voice data. Furthermore, there are no systems that can effectively respond to situations requiring real-time voice responses that reflect the user's emotions. This leads to insufficient customer service in virtual stores and a poor user experience.

[1577] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing user voice data and acquiring text data and emotion data, means for generating natural voice data from the text data and emotion data, and means for assigning a unique identifier to the generated voice data and storing it in a database. This makes it possible to generate natural and diverse voice data that reflects emotions, and realizes real-time user responses in virtual stores, etc.

[1578] "Means for recording the user's voice" refers to a device or software for recording the user's spoken voice as digital data.

[1579] "Means for transmitting recorded voice data to a server" refers to a device or software that has the function of transferring recorded voice data to a server via a network.

[1580] "Means for analyzing voice data received by the server and converting it into text data" refers to technology for processing voice data received by the server and converting it into text in natural language.

[1581] "Means for obtaining emotional data from analyzed text data" refers to technology that uses analyzed text data to recognize and determine the emotional state of the speaker.

[1582] "Means for generating voice data from text data and emotional data" refers to technology that creates natural, emotionally rich voice data based on text data and emotional data.

[1583] "Means for assigning an identifier to generated voice data" refers to a technology for adding unique identification information to generated voice data to prove the authenticity and uniqueness of the data.

[1584] "Means for storing voice data with identifiers in a database" refers to a technology for securely storing voice data with unique identification information added in a database.

[1585] "Means for listing voice data with identifiers on a marketplace" refers to technology that makes voice data with identifiers attached publicly available on an online platform and makes it available for sale or trading.

[1586] "Means for managing transactions of voice data in a marketplace" refers to technology that oversees the buying, selling, and exchange of voice data on a marketplace and ensures that transactions are carried out smoothly.

[1587] "Means for downloading audio data purchased by a user to a terminal" refers to technology for saving audio data purchased by a user on a marketplace to the user's device.

[1588] "Means for playing downloaded audio data on a terminal" refers to software or hardware for playing audio data downloaded to a terminal.

[1589] "Means for generating and instantly playing back audio data in real time" refers to technology for instantly generating audio data based on audio or text input by a user and playing it back in real time.

[1590] The system for implementing this invention comprises a voice recording means, a server communication means, a voice analysis and generation means, an identifier assignment means, a database storage means, a marketplace operation means, a download means, a playback means, a real-time voice generation means, and an emotion analysis means.

[1591] User processing

[1592] The user launches a dedicated application using a smartphone or head-mounted display (HMD). The application provides "Start Recording" and "Stop Recording" buttons. The user operates these buttons to record audio, and then presses the "Send" button to send the recorded audio data to the server. At this time, the audio file is automatically uploaded to the server via the Internet.

[1593] Server-side processing

[1594] The server first temporarily stores the received voice data. It then converts the voice data into text data using the Google Cloud Speech-to-Text API. At the same time, it analyzes emotion data from the text data using the Emotion API. It then uses the Google Cloud Text-to-Speech API to generate natural-sounding voice data based on the text data and emotion data. The generated voice data is assigned an identifier using an NFT (non-fungible token) and stored in a database.

[1595] Marketplace operations

[1596] The server also includes a function for listing voice data with identifiers on the marketplace. Users can access the marketplace and list or purchase voice data. The server is also responsible for setting prices and managing transactions for voice data on the marketplace. Users who purchase voice data can download the voice data to their own devices via the cloud service.

[1597] Terminal side processing

[1598] The downloaded voice data can be played back on the user's device within a dedicated application. It is also possible to generate voice based on text data entered in real time and play it back instantly. By providing natural, rich voice responses that reflect emotion analysis, customer service in virtual stores is enhanced.

[1599] Examples of specific examples and prompts

[1600] For example, consider the case where a user inquires about trying on clothes in a virtual store. When the user says, "Do you prefer this outfit in black or red?", the voice is recorded and sent to the server. The server analyzes the voice and obtains text data and emotion data. A voice response generated based on this data, "Black clothes are chic and lovely," is sent to the user's device and played back.

[1601] Example prompt for generative AI model:

[1602] "Write a Python program that converts the user's voice data into text, analyzes the emotion data, and then generates the speech again. Use the Google Cloud Speech-to-Text and Google Cloud Text-to-Speech APIs, and the Emotion API for emotion analysis."

[1603] This invention makes it possible to generate natural and diverse voice data that reflects emotions, thereby enhancing customer service in virtual stores, etc. This is expected to improve the user experience.

[1604] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1605] Step 1:

[1606] The user starts the dedicated application and clicks the "Start Recording" button to record audio.

[1607] Input: User utterance

[1608] Output: Recorded audio data

[1609] Specific operation: Records the user's voice using the microphone built into the smartphone or head-mounted display (HMD) and saves the audio file in local storage.

[1610] Step 2:

[1611] The user clicks the "End Recording" button and then presses the "Send" button to send the recorded voice data to the server.

[1612] Input: Recorded audio data

[1613] Output: Audio data sent to the server

[1614] Specific behavior: A process is executed to upload an audio file to a server via the Internet.

[1615] Step 3:

[1616] The server temporarily stores the received audio data.

[1617] Input: Audio data sent to the server

[1618] Output: Temporarily saved audio data

[1619] What it does: Saves the audio file to the server's local storage or temporary cache area.

[1620] Step 4:

[1621] The server uses the Google Cloud Speech-to-Text API to analyze the audio data and convert it into text data.

[1622] Input: Temporarily stored audio data

[1623] Output: Parsed text data

[1624] Specific operation: Sends an audio file to the API and retrieves the resulting string data as text data.

[1625] Step 5:

[1626] The server uses the Emotion API to obtain emotion data from text data.

[1627] Input: Parsed text data

[1628] Output: Emotion data

[1629] What it does: Sends text data to the Emotion API and retrieves data indicating the emotional state based on that text.

[1630] Step 6:

[1631] The server uses the Google Cloud Text-to-Speech API to generate voice data based on the text data and emotion data.

[1632] Input: Text data and emotion data

[1633] Output: Generated audio data

[1634] What it does: Sends text and emotion data to the API and retrieves the generated audio file.

[1635] Step 7:

[1636] The server assigns an NFT identifier to the generated audio data and stores it in a database.

[1637] Input: Generated audio data

[1638] Output: Audio data with identifiers

[1639] What it does: Adds a unique identifier to the audio file and stores it in a database.

[1640] Step 8:

[1641] The server puts the voice data with the identifier on the marketplace.

[1642] Input: Audio data with identifiers

[1643] Output: Audio data listed on the marketplace

[1644] What it does: Publishes audio files through the marketplace's management system, making them accessible to other users.

[1645] Step 9:

[1646] When a user purchases audio data from the marketplace, the data is downloaded to the device.

[1647] Input: Audio data selected from the marketplace

[1648] Output: Audio data downloaded to the device

[1649] Specific operation: Transfer data from cloud storage to the user's device and save it in local storage.

[1650] Step 10:

[1651] The terminal plays the downloaded audio data.

[1652] Input: Audio data downloaded to the device

[1653] Output: The audio data to be played

[1654] Specific Behavior: Plays an audio file using the audio player functionality within the application.

[1655] Step 11:

[1656] The device generates audio based on the text data entered in real time and plays it back instantly.

[1657] Input: Text data entered by the user

[1658] Output: Audio data generated and played in real time

[1659] Specific operation: Sends text data to the server, receives real-time generated audio data, and plays it back immediately.

[1660] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1661] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1662] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1663] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1664] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1665] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1666] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1667] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1668] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1669] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1670] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1671] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1672] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1673] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1674] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1675] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1676] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1677] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1678] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1679] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1680] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1681] The following is further disclosed regarding the above embodiment.

[1682] (Claim 1)

[1683] a means for recording the user's voice;

[1684] means for transmitting the recorded voice data to a server;

[1685] A means for analyzing the received voice data in the server and converting it into text data;

[1686] means for generating voice data from text data;

[1687] means for assigning an identifier to the generated voice data;

[1688] means for storing the identified voice data in a database;

[1689] A means for listing the identified voice data on a marketplace;

[1690] a means for managing transactions of audio data in the marketplace;

[1691] A means for downloading the audio data purchased by the user to the terminal;

[1692] means for playing audio data;

[1693] A system including:

[1694] (Claim 2)

[1695] 10. The system of claim 1, further comprising means for setting a price for the audio data that the user sells in the marketplace.

[1696] (Claim 3)

[1697] 10. The system according to claim 1, further comprising means for generating and playing back voice data in real time based on text data entered by a user.

[1698] "Example 1"

[1699] (Claim 1)

[1700] a means for recording the user's voice;

[1701] means for transmitting the recorded voice data to a data server;

[1702] A means for analyzing the received voice data in the data server and converting it into text data;

[1703] means for generating voice data from text data;

[1704] means for assigning an identifier to the generated voice data;

[1705] means for storing the identified voice data in a database;

[1706] means for listing the identified audio data on an electronic marketplace;

[1707] means for managing trading of voice data in an electronic marketplace;

[1708] means for downloading the audio data purchased by the user to the end user terminal;

[1709] means for playing audio data;

[1710] A system including:

[1711] (Claim 2)

[1712] 10. The system of claim 1, further comprising means for setting a price for the audio data sold by the user on the electronic marketplace.

[1713] (Claim 3)

[1714] 10. The system according to claim 1, further comprising means for generating and playing back voice data in real time based on text data entered by a user.

[1715] "Application Example 1"

[1716] (Claim 1)

[1717] a means for recording the user's voice;

[1718] means for transmitting the recorded voice data to a server;

[1719] A means for analyzing the received voice data in the server and converting it into text data;

[1720] means for generating voice data from text data;

[1721] means for assigning an identifier to the generated voice data;

[1722] means for storing the identified voice data in a database;

[1723] A means for listing the identified voice data on a marketplace;

[1724] a means for managing transactions of audio data in the marketplace;

[1725] A means for downloading the audio data purchased by the user to the terminal;

[1726] means for playing audio data;

[1727] a means for performing authentication using recorded voice data;

[1728] a means for matching the generated voice data to improve authentication accuracy;

[1729] A system including:

[1730] (Claim 2)

[1731] 10. The system of claim 1, further comprising means for setting a price for the audio data that the user sells in the marketplace.

[1732] (Claim 3)

[1733] 10. The system according to claim 1, further comprising means for generating and playing back voice data in real time based on text data entered by a user.

[1734] "Example 2: Combining Emotion Engines"

[1735] (Claim 1)

[1736] a means for recording the user's voice;

[1737] means for transmitting the recorded voice data to a server;

[1738] A means for analyzing the received voice data in the server and converting it into text data;

[1739] means for generating voice data from text data;

[1740] means for assigning an identifier to the generated voice data;

[1741] means for storing the identified voice data in a database;

[1742] A means for listing the identified voice data on a marketplace;

[1743] a means for managing transactions of audio data in the marketplace;

[1744] A means for downloading the audio data purchased by the user to the terminal;

[1745] means for playing audio data;

[1746] means for recognizing emotions from user voice data and reflecting the emotions in generated voice data;

[1747] A means for storing multiple voice data entered by the user in local storage and backing it up to the cloud as needed;

[1748] A system including:

[1749] (Claim 2)

[1750] 10. The system of claim 1, further comprising means for setting a price for the audio data that the user sells in the marketplace.

[1751] (Claim 3)

[1752] 10. The system according to claim 1, further comprising means for generating and playing back voice data in real time based on text data entered by a user.

[1753] "Application example 2 when combining emotion engines"

[1754] (Claim 1)

[1755] a means for recording the user's voice;

[1756] means for transmitting the recorded voice data to a server;

[1757] A means for analyzing the received voice data in the server and converting it into text data;

[1758] A means for obtaining emotion data from the analyzed text data;

[1759] means for generating voice data from text data and emotion data;

[1760] means for assigning an identifier to the generated voice data;

[1761] means for storing the identified voice data in a database;

[1762] A means for listing the identified voice data on a marketplace;

[1763] a means for managing transactions of audio data in the marketplace;

[1764] A means for downloading the audio data purchased by the user to the terminal;

[1765] means for playing the downloaded audio data on the terminal;

[1766] a means for generating and instantly playing back audio data in real time;

[1767] A system including:

[1768] (Claim 2)

[1769] 10. The system of claim 1, further comprising means for setting a price for the audio data that the user sells in the marketplace.

[1770] (Claim 3)

[1771] 2. The system according to claim 1, further comprising means for generating voice data in real time based on text data input by a user and playing back the voice data while reflecting emotion data. [Explanation of symbols]

[1772] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for recording the user's voice; means for transmitting the recorded voice data to a server; A means for analyzing the received voice data in the server and converting it into text data; means for generating voice data from text data; means for assigning an identifier to the generated voice data; means for storing the identified voice data in a database; A means for listing the identified voice data on a marketplace; a means for managing transactions of audio data in the marketplace; A means for downloading the audio data purchased by the user to the terminal; means for playing audio data; A system including:

2. 10. The system of claim 1, further comprising means for setting a price for the audio data that the user sells in the marketplace.

3. 2. The system according to claim 1, further comprising means for generating and reproducing voice data in real time based on text data entered by a user.

Images