system

Abstract

Provide a system. A registration means for registering voice data of a specific person; a rights management means for managing rights to use the audio data; a request receiving means for receiving a synthesis request; a voice synthesis means for synthesizing voice data based on a synthesis request; providing means for providing synthesized voice data; a monitoring means for monitoring the usage of the voice data; detection means for detecting unauthorized use of the audio data; a notification means for providing an alert when unauthorized use is detected; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Advances in speech synthesis AI have made it easier to generate realistic speech based on a person's voice. However, there is an increased risk of unauthorized use of voice data, and problems have arisen in which voice providers are not paid fair compensation. Another issue is the difficulty of accurately managing and synthesizing voice data to provide new value. The present invention aims to provide a platform that protects the rights of voice data providers and prevents unauthorized use, while providing new service value using speech synthesis technology. [Means for solving the problem]

[0005] The present invention provides a system for registering voice data of a specific person, managing rights to use the voice data, accepting synthesis requests, and synthesizing voice data based on the synthesis requests. This system includes a registration means, rights management means, request acceptance means, voice synthesis means, provision means, monitoring means, detection means, and notification means. It also includes a profit distribution means for distributing profits for the use of the voice data to data providers, and uses a voice synthesis AI model to realistically reproduce the voice of the specific person. This protects the rights of voice data providers and provides new valuable services.

[0006] The "registration means" is a system element that has the function of recording the voice data of a specific person, inputting necessary metadata, and transmitting the data to the server.

[0007] The "rights management means" is a system element that has the function of managing the usage conditions set by the audio data provider and storing them in a database.

[0008] The "request receiving means" is a system element that provides an interface for a data user to create a request for synthesis of voice data and send it to the server.

[0009] A "voice synthesis means" is a system element that has the function of synthesizing input text into realistic voice using voice data of a specific person based on a synthesis request.

[0010] The "means for providing" is a system element that has the function of providing synthesized voice data to data users and storing it in a database.

[0011] "Monitoring means" means a system element that has the function of continuously monitoring the usage of voice data and detecting signs of unauthorized use.

[0012] "Detection means" refers to a system element that has the function of detecting unauthorized use or illegal access to audio data.

[0013] "Notification means" refers to a system element that has the function of notifying data providers and administrators of an alert when unauthorized use is detected.

[0014] A "profit sharing means" is an element of the system that has the function of calculating profits for the use of voice data and distributing them to data providers.

[0015] A "speech synthesis AI model" is a model that uses artificial intelligence technology to convert input text into realistic speech based on the voice data of a specific person. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system for providing a speech synthesis service based on the speech data of a specific person, and includes elements of a user, a terminal, and a server. Below, we will explain how the elements of this system work together to realize specific functions.

[0038] 1. Registering audio data

[0039] The user (voice data provider) installs a dedicated app on their device to record their voice. The user launches the app and records the voice to be provided. They also enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality).

[0040] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data and stores it in a database. It extracts features of the audio data (e.g., pitch, tone, and sound quality), generates identification information, and records it in the database.

[0041] 2. Rights Management

[0042] Users log in to the platform and set the terms of use for their own voice data (whether commercial use is permitted, usage fees, etc.). The device displays a form for entering this setting information and sends the user's input to the server.

[0043] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[0044] 3. Accepting synthesis requests

[0045] A user (data user) makes a request through the platform to generate audio based on a specific person's voice. The device provides a form for the user to select the provider of the audio data they want to use and to input the content they want to generate (such as a text message or lyrics). The user enters the request details, and the device sends this information to the server.

[0046] 4. Speech Synthesis

[0047] The server retrieves the corresponding voice data from the database based on the received synthesis request. Using a voice synthesis AI model, it inputs the text provided by the user and generates a realistic voice in the voice of a specific person. The synthesized voice data is stored in the database on the server.

[0048] 5. Provision of audio data

[0049] The server provides the generated voice data to the data user's terminal. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[0050] 6. Monitoring and Alerts

[0051] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[0052] Specific examples

[0053] For example, voice data provider A records his or her voice and registers it on the platform. A sets conditions allowing only commercial use. Meanwhile, advertising production company B wants to create a narration for a product introduction using provider A's voice and sends a request. The synthetic AI generates a narration in provider A's voice saying, "Please try this product," and provides it to B. The generated voice is for commercial use, and the compensation is distributed to provider A. During this process, the server monitors all usage and checks for any unauthorized use.

[0054] This system protects the rights of voice data providers and ensures appropriate profit distribution. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to users.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[0058] Step 2:

[0059] The user presses the record button to record their own voice. After the recording is complete, the device temporarily stores the recorded voice data and displays a form for inputting necessary metadata (such as name, age, gender, and voice description).

[0060] Step 3:

[0061] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[0062] Step 4:

[0063] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[0064] Step 5:

[0065] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[0066] Step 6:

[0067] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[0068] Step 7:

[0069] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[0070] Step 8:

[0071] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[0072] Step 9:

[0073] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[0074] Step 10:

[0075] The server receives the request, retrieves the corresponding voice data from a database, and then uses a speech synthesis AI model to input the specified text and generate speech in the voice of a specific person.

[0076] Step 11:

[0077] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[0078] Step 12:

[0079] The server calculates profits based on the amount of voice data used, and the calculated profits are distributed to the data providers and recorded in a database.

[0080] Step 13:

[0081] The server continuously monitors the usage of voice data to detect signs of unauthorized use and generates an alert if an abnormality is detected.

[0082] Step 14:

[0083] The server sends the generated alert to the data provider's terminal, which displays the alert notification so that the user can check the status.

[0084] Through the above steps, this system enables the appropriate management and utilization of voice data, protecting the rights of data providers and providing new value.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] In speech synthesis systems based on voice data, there is a need to efficiently and safely manage the voice data of specific individuals, prevent unauthorized use, and realize accurate and realistic voice synthesis. In addition, a mechanism is needed to protect the rights of voice data providers and distribute profits appropriately.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing rights to use the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a provision means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, a means for encrypting and transmitting metadata associated with the recorded voice data, and a prompt sentence generation means for inputting the voice data and text into a voice synthesis model. This enables safe management and use of voice data, protection of the rights of voice data providers, and realization of realistic voice synthesis.

[0090] The "registration means" is a mechanism for recording the voice data of a specific person and registering it in a database together with necessary metadata.

[0091] The "rights management means" is a mechanism for managing the rights to use audio data, and for setting and updating the conditions of use and permission details.

[0092] The "request receiving means" is a mechanism for receiving a request for voice synthesis based on voice data and transmitting the content of the request to the server.

[0093] The "voice synthesis means" is a mechanism for generating voice using voice data of a specific person based on a synthesis request.

[0094] The "means for providing" is a mechanism for providing the generated voice data to the terminal of the data user.

[0095] "Monitoring measures" are mechanisms for continuously monitoring the usage of voice data and detecting fraudulent or unauthorized use.

[0096] The "detection means" is a mechanism for determining whether or not there is unauthorized use of audio data based on information from the monitoring means.

[0097] The "notification means" is a mechanism for notifying relevant parties of an alert when unauthorized use of audio data is detected.

[0098] "Means for encryption and transmission" refers to a mechanism for encrypting recorded audio data and associated metadata and transmitting them to a server using a secure communication protocol.

[0099] The "prompt sentence generation means" is a mechanism for generating a prompt sentence for inputting voice data and text into a voice synthesis model.

[0100] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and includes elements of a user, a terminal, and a server. This system has the necessary configuration to realize the functions of voice data registration, rights management, reception of synthesis requests, voice synthesis, voice data provision, monitoring, and alerts.

[0101] Users install a dedicated app on their device to record their own voice. Using this app, users record their voice and enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality). The device encrypts the recorded voice data and the entered metadata and sends them to the server using a security protocol such as TLS (Transport Layer Security). The server analyzes the received voice data using acoustic analysis software (e.g., Praat or Audacity), extracts voice features, and stores them in a database.

[0102] Next, the user logs in to the platform and sets the conditions for using the audio data (whether commercial use is permitted, usage fees, etc.). The terminal displays a form for entering this setting information and sends the user's input to the server. The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions.

[0103] When generating audio using audio data, the user (data user) selects an audio data provider through the platform and inputs the content they want to generate (such as a text message or lyrics). For example, they input the text "Please try this product." The device then sends this information to the server.

[0104] The server retrieves the corresponding voice data from the database and inputs the text provided by the user using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). Specifically, the prompt "Try this product" is input into the speech synthesis model. The server stores the generated voice data in the database and then provides it to the user's device. The user can download the provided voice data and use it for the specified purpose.

[0105] Furthermore, the server continuously monitors the usage of voice data to detect signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or behavior that violates usage policies, it generates an alert and notifies the data provider's device, allowing the user to take appropriate action.

[0106] For example, if voice data provider A records his or her voice and registers on the platform under the condition that only commercial use is permitted, and an advertising production company wishes to create a narration in A's voice saying, "Please try this product." The server that receives this request through the platform uses A's voice to generate a narration using a voice synthesis model and provides it to the advertising production company. The generated voice is used commercially, and the compensation is distributed to A.

[0107] This system protects the rights of voice data providers and ensures appropriate profit sharing. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to each user.

[0108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0109] Step 1: Record audio and enter metadata

[0110] The user installs a dedicated recording app on their device and launches it. The user presses the record button to record their own voice. Once recording is complete, the app displays a form in which the user can enter profile information (name, age, gender, voice quality, etc.). The data entered by the user here (audio data and metadata) is the input. The recorded audio data and the entered metadata are generated as the output.

[0111] Step 2: Encrypt and transmit audio data and metadata

[0112] The device encrypts the recorded audio data and input metadata using a security protocol such as TLS (Transport Layer Security). The encrypted audio data and metadata are input, and the output is sent to a server via a secure communication protocol.

[0113] Step 3: Analyze and save the audio data

[0114] The server decrypts the received encrypted audio data and metadata and analyzes it using acoustic analysis software (e.g., Praat or Audacity). This analysis extracts audio features (pitch, tone, timbre, etc.). The input is the decrypted audio data, and the output is the analyzed audio features and a process to store them in a database.

[0115] Step 4: Set the terms of use

[0116] The user logs in to a web or mobile platform and sets the terms of use for the audio data (whether commercial use is permitted, the usage fee, etc.). The terminal displays a form for inputting this setting information and accepts the user's input. The set terms of use are the input, and the terminal sends this information to the server. As an output, the rights management information in the database is updated.

[0117] Step 5: Accepting a synthesis request

[0118] The user (data user) selects a voice data provider and inputs the text of the content they want to generate. The input is the input text (e.g., a prompt sentence such as "Try this product"). The terminal sends this information to the server. The server receives a synthesis request as output.

[0119] Step 6: Acquiring audio data and synthesizing speech

[0120] The server retrieves the corresponding voice data from the database and inputs the text using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). The inputs are the retrieved voice data and the prompt. The speech synthesis AI model uses these to generate realistic speech. The output is the synthesized voice data, which is stored in the database.

[0121] Step 7: Provide synthetic voice data

[0122] The server provides the generated voice data to the user's (data user's) terminal. The provided voice data is the output. The user downloads this voice data and uses it for the specified purpose.

[0123] Step 8: Monitor usage and generate alerts

[0124] The server monitors the usage of voice data in real time. The input is the database access log. If the monitoring detects any fraudulent or unauthorized use, an alert is generated. The generated alert (notification message) is the output.

[0125] Step 9: Alert Notification

[0126] The server notifies the generated alert to the user's (data provider's) device. The alert notification (for example, app notification or email) is the output, allowing the user to take action against fraudulent use.

[0127] Through this series of steps, safe management and use of voice data, protection of the rights of voice data providers, and realistic voice synthesis are realized.

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] In conventional virtual stores, when users ask questions about products or services in real time, only text-based responses are provided, which poses a problem of lacking realism and reliability. Furthermore, even when using speech synthesis technology, there is a lack of a mechanism for reproducing a specific person's voice in real time, which limits the interactive experience. For this reason, there is a need for a system that can respond in real time using a specific person's voice.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes: a registration means for registering voice data of a specific person; a rights management means for managing usage rights of the voice data; a request receiving means for receiving a synthesis request; a voice synthesis means for synthesizing voice data based on the synthesis request; a provision means for providing the synthesized voice data; a monitoring means for monitoring the usage status of the voice data; a detection means for detecting unauthorized use of the voice data; a notification means for issuing an alert when unauthorized use is detected; a generation AI model utilization means for using a generation AI model when a user uses the voice synthesis service and synthesizing voice data in real time based on a prompt sentence; and a voice data provision means for providing the synthesized voice data to a terminal and receiving the voice data as a voice file, thereby enabling a user to receive explanations about products and services in real time in a virtual store from the voice of a specific person.

[0133] "Voice data of a specific person" is data that is a digital recording of a voice provided by a specific individual.

[0134] The "registration means" is a function that allows voice data providers to register their own voice data in the system.

[0135] The "rights management means" is a function that manages the conditions of use and rights information for audio data, and protects the rights of data providers.

[0136] The "request receiving means" is a function that allows a user of voice data to request voice synthesis using the voice of a specific person.

[0137] The "voice synthesis means" is a function for converting designated text into voice by using voice data of a specific person based on a request.

[0138] The "provision means" is a function for providing the generated voice data to the user.

[0139] "Monitoring means" is a function that monitors the usage of voice data and checks for any fraudulent or unauthorized use.

[0140] "Detection means" is a function for detecting unauthorized or improper use.

[0141] The "notification means" is a function for notifying the data provider of an alert when unauthorized use is detected.

[0142] "Means for utilizing a generative AI model" refers to a function for synthesizing voice data in real time based on a prompt sentence using a generative AI model.

[0143] The "voice data providing means" is a function that provides synthesized voice data to the terminal and allows the user to receive the voice data through a voice file.

[0144] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and is composed of a user, a terminal, and a server. This system includes functions for registering the voice data of a specific person, managing rights, accepting synthesis requests, voice synthesis, providing voice data, monitoring, detecting and notifying unauthorized use, utilizing a generative AI model, and providing data.

[0145] 1. Registering audio data

[0146] Users install an application on their device to record their own voice. They use this application to record voice data and enter metadata such as profile information and sound quality. The device then sends the recorded voice data and metadata to a server. The server analyzes the voice data and stores it in a database.

[0147] 2. Rights Management

[0148] The user logs in to the server and sets the conditions for using their own audio data. The terminal sends this information to the server. The server updates the rights management information in the database based on the condition information and generates a usage policy.

[0149] 3. Accepting synthesis requests

[0150] A user of voice data makes a request through a terminal to generate voice data based on the voice of a specific person. The terminal then selects the voice data provider they wish to use and provides a form for inputting the text message or question they wish to generate. This information is then sent to the server.

[0151] 4. Speech Synthesis

[0152] The server retrieves the corresponding voice data from the database based on the received synthesis request, and uses the generative AI model to synthesize a realistic voice with the voice of a specific person based on the prompt sentence. The generated voice data is stored in the database on the server.

[0153] 5. Provision of audio data

[0154] The server provides the generated voice data to the terminal, which the user can download and use for the specified purpose.

[0155] 6. Monitoring and Alerts

[0156] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. If unauthorized use is detected, the server generates an alert and notifies the data provider's device.

[0157] Specific examples

[0158] For example, if a user asks, "Please tell me about the ingredients in this new cosmetic product," the server uses a generative AI model to generate a response text. Based on this text, the server uses a speech synthesis API such as AWS (registered trademark) Polly to generate voice data and respond in the voice of a specific person. This voice data is then provided to the user's device.

[0159] Example prompt sentence:

[0160] "Please tell me about the ingredients in this new cosmetic product."

[0161] In this way, by using this system, users can enjoy a realistic voice interaction experience and receive explanations about products and services in the virtual store.

[0162] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0163] Step 1:

[0164] Users use a dedicated application installed on their device to record their own voice and input metadata such as profile information and sound quality. Input: Voice data, profile information. Output: Recorded voice data, metadata.

[0165] Step 2:

[0166] The device sends the recorded audio data and input metadata to the server. Input: Recorded audio data, metadata. Output: Audio data and metadata sent to the server.

[0167] Step 3:

[0168] The server analyzes the received voice data and stores it in a voice database. At the same time, it extracts features from the voice data and records them in the database. Input: Voice data, metadata. Output: Stored voice data, features.

[0169] Step 4:

[0170] The user logs in to the server and sets the conditions for using their own audio data on the rights management screen. Input: Conditions of use (whether commercial use is permitted, usage fees, etc.). Output: The set conditions of use.

[0171] Step 5:

[0172] The terminal transmits the set usage conditions to the server, which records them in the rights management database. Input: Usage conditions. Output: Usage conditions recorded in the rights management database.

[0173] Step 6:

[0174] The user of the voice data makes a request through their terminal to generate voice based on the voice of a specific person. In the form, they enter the voice data provider they want to use and the text message they want to generate. Input: Voice data provider, text message. Output: Request details.

[0175] Step 7:

[0176] The terminal sends the request content to the server, and the server retrieves the corresponding voice data from the database. Input: Request content. Output: Retrieved voice data.

[0177] Step 8:

[0178] The server uses a generative AI model to convert text to speech in a specific person's voice based on the prompt and the acquired voice data. Input: Prompt, voice data. Output: Generated voice data.

[0179] Step 9:

[0180] The server stores the generated voice data in a database and provides it to the user's device. Input: Generated voice data. Output: Provided voice data.

[0181] Step 10:

[0182] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. Input: Usage data. Output: Monitoring reports, alerts.

[0183] Step 11:

[0184] If unauthorized use is detected, the server generates an alert and notifies the data provider's terminal. Input: Detection result. Output: Alert notification.

[0185] Step 12:

[0186] The user receives the voice data generated in real time through the terminal and uses it as an audio file. For example, if the question is "Please tell me about the ingredients of this new cosmetic product," the user will receive the answer in real time in a specific voice. Input: Provided voice data. Output: Playback of the audio file.

[0187] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0188] The present invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person, and its purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we will explain how the elements of this system work together to realize specific functions.

[0189] 1. Registering audio data

[0190] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input. The user presses the record button to record their own voice and enters the necessary metadata (name, age, gender, voice description, etc.).

[0191] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data, extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[0192] 2. Rights Management

[0193] The user logs in to the platform and sets the terms of use for their own voice data. The terminal displays a form that prompts the user to enter the terms of use (for example, whether commercial use is permitted, the usage fee, etc.). When the user enters the terms of use and presses the send button, the terminal sends the entered terms of use to the server.

[0194] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[0195] 3. Synthesis request acceptance and emotion recognition

[0196] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal provides a form in which the user selects a provider of the voice data they wish to use and inputs the content they wish to generate (e.g., text message or lyrics). The terminal then activates an emotion recognition means to recognize emotions from the user's input data and analyzes the emotions from the input text.

[0197] 4. Speech synthesis and emotion-based adjustment

[0198] The emotion recognition means analyzes emotions (e.g., joy, sadness, anger, etc.) from the input text and sends the emotion to the server, which retrieves the corresponding voice data from the database and generates voice using a voice synthesis AI model based on the user's emotion.

[0199] For example, if the input text is "I'm so happy today!", the emotion recognition means will determine this as "joy" and provide that information to the AI ​​model, which will then generate speech with intonation and nuances that match the emotion of joy.

[0200] 5. Provision of audio data

[0201] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[0202] 6. Monitoring and Alerts

[0203] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[0204] Specific examples

[0205] For example, voice data provider A records his or her voice and registers it on the platform. A sets a condition that only commercial use is permitted. Meanwhile, advertising production company B wants to create a narration for a product introduction in provider A's voice and sends a request. If the text provided by user B is "Please try this product," the emotion recognition means analyzes the text and recognizes a friendly emotion. Based on this emotional information, the server generates a narration in provider A's voice with a friendly tone. The generated voice is provided to advertising production company B, and the compensation is distributed to provider A.

[0206] This system protects the rights of voice data providers and enables high-quality voice synthesis that reflects emotions, thereby providing new value.

[0207] The processing flow will be explained below.

[0208] Step 1:

[0209] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[0210] Step 2:

[0211] The user presses the record button to record their own voice. Once the recording is complete, the device temporarily stores the recorded voice data and displays a form for entering necessary metadata (such as name, age, gender, and voice description).

[0212] Step 3:

[0213] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[0214] Step 4:

[0215] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[0216] Step 5:

[0217] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[0218] Step 6:

[0219] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[0220] Step 7:

[0221] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[0222] Step 8:

[0223] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[0224] Step 9:

[0225] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[0226] Step 10:

[0227] The server receives the request and retrieves the corresponding voice data from the database. Then, it uses an emotion recognition unit to analyze emotions based on the request text. The emotion recognition unit analyzes keywords and context within the text to identify specific emotions (e.g., joy, sadness, anger, etc.).

[0228] Step 11:

[0229] The server uses a speech synthesis AI model based on the analyzed emotional information to generate an appropriate voice from the voice data of a specific person. For example, if the text says "I'm so happy today!", the emotion recognition means will identify "joy" and the speech synthesis AI will generate a voice that expresses "joy."

[0230] Step 12:

[0231] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[0232] Step 13:

[0233] The server calculates profits based on the amount of voice data used, and records the calculated profits to be distributed to users (data providers).

[0234] Step 14:

[0235] The server continuously monitors voice data usage to detect signs of abuse and generates an alert if unauthorized use is detected.

[0236] Step 15:

[0237] The server sends the generated alert to the user's (data provider's) terminal, which displays the alert notification so that the user can check the status.

[0238] By combining emotion recognition and speech synthesis technologies, this system can generate and provide high-quality audio content that matches the user's emotions, thereby protecting the rights of audio data providers.

[0239] Example 2

[0240] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0241] Conventional speech synthesis systems have difficulty synthesizing a specific person's voice naturally, and speech synthesis based on emotion recognition in particular has not been fully realized. Furthermore, managing the rights to use voice data and monitoring and detecting unauthorized use are difficult, and the rights of data providers are not adequately protected. Therefore, there is a need for a system that not only provides high-quality speech synthesis but also reliably protects the rights of data providers.

[0242] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0243] In this invention, the server includes a registration means for registering voice data of a specific person through a dedicated application installed on the user's device, an analysis means for analyzing the voice data and extracting features, a rights management means for allowing the user to set usage conditions for the voice data, a voice synthesis means for analyzing emotions from input text using an emotion recognition means and synthesizing voice based on the emotions, a provision means for storing the synthesized voice data in a database and providing it, a monitoring means for continuously monitoring the usage status of the voice data, and a notification means for detecting unauthorized use of the voice data and notifying the user. This not only enables high-quality voice synthesis and natural-sounding voice generation based on the user's emotions, but also protects the rights of data providers and makes it easy to monitor and detect unauthorized use.

[0244] A "dedicated app" is software with specific functions that is installed on a user's device and used.

[0245] "Voice data" refers to acoustic information that is a recording of a specific person's voice and stored in digital format.

[0246] "Features" are characteristic values ​​such as pitch, tone, and sound quality extracted from audio data.

[0247] "Rights management" refers to the procedures and systems for setting and managing the terms of use and rights information for audio data.

[0248] An "emotion recognition means" is a technology or tool that analyzes and identifies a user's emotions from input text or voice.

[0249] "Speech synthesis means" refers to technology and devices that generate new voices based on specific voice data and emotional information.

[0250] "Providing means" refers to the process and system for providing synthesized voice data to the user.

[0251] "Monitoring Measures" are technologies and tools that continuously monitor the usage of voice data.

[0252] The "notification means" refers to a system and method for notifying the user when unauthorized use or abnormality of audio data is detected.

[0253] The "profit distribution means" is a method and system for calculating profits according to the amount of use of voice data and distributing them to data providers.

[0254] A "voice synthesis AI model" is a model that uses artificial intelligence to generate high-quality voice based on voice data and emotional information.

[0255] This invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person. Its particular purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we explain how the elements of this system work together to realize specific functions.

[0256] 1. Registering audio data

[0257] The user installs a dedicated app on their device and launches it. The app has an interface for recording audio, and the user presses the record button to begin recording. When recording is complete, the user enters metadata such as their name, age, gender, and a description of the audio. The device then sends the recorded audio data and the entered metadata to a server. The server then analyzes the received audio data using audio analysis software (e.g., Praat, LibROSA), extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[0258] Examples:

[0259] When the user presses the record button, the device starts recording.

[0260] After recording, the user enters the metadata and sends it to the server.

[0261] The server analyzes the voice data and stores the features in a database.

[0262] 2. Rights Management

[0263] The user logs in to the platform and sets the terms of use for the audio data. The device displays a form for entering the terms of use (e.g., whether commercial use is permitted, and the usage fee). The user enters the terms of use, and the device sends the information to the server. The server updates the rights management information in the database based on the received terms and generates a usage policy based on the terms of use.

[0264] Examples:

[0265] The user logs into the platform and enters the terms of use.

[0266] The terminal transmits the entered usage conditions to the server.

[0267] The server generates a usage policy and stores it in a database.

[0268] 3. Synthesis request acceptance and emotion recognition

[0269] The user (data user) accesses the platform and creates a request for speech synthesis. The device provides a form where the user can select the provider of the voice data they wish to use and input the content they wish to generate (e.g., text message or lyrics). The device sends the input text to emotion recognition software (e.g., Google® Cloud Natural Language, IBM Watson®), which analyzes emotions from the text.

[0270] Examples:

[0271] The user inputs the content they want to generate.

[0272] The device sends the input text to emotion recognition software, which analyzes the emotion.

[0273] 4. Speech synthesis and emotion-based adjustment

[0274] The emotion recognition software analyzes the emotional information and sends it to the server, which retrieves the corresponding voice data from the database and generates speech using a speech synthesis AI model (e.g., Tacotron, WaveNet). The generated speech is adjusted according to the emotion.

[0275] Examples:

[0276] The server acquires the emotion information and acquires the voice data from the database.

[0277] The server uses a speech synthesis AI model to generate speech that corresponds to the emotion.

[0278] 5. Provision of audio data

[0279] The server stores the generated voice data in a database and provides it to the data user's device. The data user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[0280] Examples:

[0281] The server stores the generated audio data in a database.

[0282] The device downloads the audio data.

[0283] Example prompt sentence:

[0284] "Using the voice of voice data provider A, synthesize the following text with a friendly emotion: 'Try this product.'"

[0285] 6. Monitoring and Alerts

[0286] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[0287] Examples:

[0288] The server monitors the access log and detects any abnormalities.

[0289] The server sends an alert notification to the user.

[0290] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0291] Step 1:

[0292] The user installs the dedicated app on their device and launches it. When the user presses the record button, the device starts recording the audio. After recording is complete, the device accepts metadata entered by the user (such as name, age, gender, and audio description).

[0293] Input: User starts recording and enters metadata

[0294] Output: Recorded audio data and metadata

[0295] Specific behavior:

[0296] The user presses the record button.

[0297] The device will record and temporarily store the audio.

[0298] The user enters the required metadata.

[0299] Step 2:

[0300] The device sends the recorded voice data and input metadata to a server, which then analyzes the received voice data using voice analysis software, extracts features (pitch, tone, timbre, etc.), and stores them in a database.

[0301] Input: Audio data and metadata sent from the device

[0302] Output: Analyzed feature data

[0303] Specific behavior:

[0304] The device encrypts the audio data and metadata and sends them to the server.

[0305] The server reads the voice data and extracts features using analysis software.

[0306] The feature data is stored in a database.

[0307] Step 3:

[0308] The user logs in to the platform and sets the terms of use for the audio data. They enter the terms of use (whether commercial use is permitted, the usage fee, etc.) in a form displayed on the device, and the device sends this information to the server. The server updates the rights management information in the database based on the received terms and conditions and generates a usage policy.

[0309] Input: Terms of Use Information

[0310] Output: Updated rights management information and usage policy

[0311] Specific behavior:

[0312] The user logs into the platform and enters the terms of use.

[0313] The terminal transmits the usage conditions to the server.

[0314] The server updates the database based on the condition information and generates a usage policy.

[0315] Step 4:

[0316] The user (data user) accesses the platform and creates a request for voice synthesis. The device provides a form where the user can select the provider of the voice data they want to use and input the content they want to generate (text message or lyrics). The device sends the input text to emotion recognition software, which analyzes emotions from the text.

[0317] Input: The content (text) you want to generate

[0318] Output: Parsed emotion information

[0319] Specific behavior:

[0320] The user selects a provider and enters text content.

[0321] The device sends the text to emotion recognition software.

[0322] Emotion recognition software analyzes emotions from text and returns emotional information.

[0323] Step 5:

[0324] The emotion recognition software analyzes the emotional information and sends it to a server, which retrieves the corresponding voice data from a database and generates a voice using a voice synthesis AI model. The generated voice is adjusted according to the emotion.

[0325] Input: Emotional information, voice data

[0326] Output: Generated audio data

[0327] Specific behavior:

[0328] The server retrieves the audio data from the database.

[0329] The server inputs emotional information and voice data into the AI ​​model.

[0330] The AI ​​model generates voice based on the emotion.

[0331] Step 6:

[0332] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[0333] Input: Generated audio data

[0334] Output: Provided audio data, profits distributed

[0335] Specific behavior:

[0336] The server stores the generated audio data in a database.

[0337] The server sends a download link for the audio data to the device.

[0338] The device downloads the audio data.

[0339] The server calculates profits based on usage and distributes them to data providers.

[0340] Step 7:

[0341] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[0342] Input: Voice data usage

[0343] Output: Alert notification

[0344] Specific behavior:

[0345] The server monitors the access log and detects any abnormalities.

[0346] The server generates an alert and notifies the user.

[0347] (Application example 2)

[0348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0349] While conventional speech synthesis technologies can generate voice data using a specific person's voice, they cannot automatically adjust the voice according to emotion, making it difficult to communicate naturally with users. Furthermore, there is a lack of effective systems for managing voice data usage rights and detecting unauthorized use. There is a need for a system that can recognize emotions and adjust the voice according to the user's text message in real time.

[0350] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing usage rights of the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a providing means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, an emotion recognition means for analyzing emotions from an input text message, and a voice synthesis adjustment means for adjusting the voice based on the analyzed emotions. This makes it possible to use the voice data of a specific person and adjust the voice based on emotions corresponding to the text message, thereby realizing more human-like communication.

[0351] The "registration means" is a function for registering voice data of a specific person in the system.

[0352] The "rights management means" is a function for appropriately managing the rights to use audio data and protecting the rights of data providers.

[0353] The "request receiving means" is a function for receiving a request to synthesize voice data.

[0354] The "voice synthesis means" is a function for generating voice data based on a synthesis request.

[0355] The "provision means" is a function for providing synthesized voice data to the user.

[0356] The "monitoring means" is a function for continuously monitoring the usage status of voice data.

[0357] The "detection means" is a function for detecting unauthorized use of audio data.

[0358] The "notification means" is a function for sending an alert to the user when unauthorized use is detected.

[0359] The "emotion recognition means" is a function for analyzing emotions from a text message entered by a user.

[0360] The "voice synthesis adjustment means" is a function for adjusting the voice data to be generated based on the analyzed emotion.

[0361] A specific system configuration and its operation will be described below for an embodiment of the present invention.

[0362] 1. Registering audio data

[0363] The user installs a dedicated application on their smartphone. They launch the application and register voice data through an interface that allows them to record their own voice. The user presses the record button, records their own voice, and enters the necessary metadata (such as name, age, gender, and voice description). The device then sends the recorded voice data and metadata to the server. The server analyzes the voice data, extracts features, and stores them in a database.

[0364] 2. Rights Management

[0365] Users log in to the platform and set the terms of use for the audio data (for example, whether commercial use is permitted, and the usage fee setting). The setting information is sent to the server, which then updates the rights management information and generates a usage policy based on that information.

[0366] 3. Synthesis request acceptance and emotion recognition

[0367] A user (data user) accesses the platform and creates a request for voice synthesis. The user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device then activates the emotion recognition means and analyzes emotions from the input text.

[0368] 4. Speech synthesis and emotion-based adjustment

[0369] The emotion recognition means analyzes the text message and determines the emotion (e.g., joy, sadness, anger, etc.). This emotion information is sent to the server, which retrieves the corresponding voice data from a database. A speech synthesis AI model is used to generate a voice corresponding to the emotion. For example, for the input text "I'm so happy today!", the system detects the emotion "joy" and generates a voice with intonation and nuance that matches the emotion of joy.

[0370] 5. Provision of audio data

[0371] The server stores the generated voice data in a database and provides it to the user's device. The user downloads the provided voice data and uses it for the specified purpose.

[0372] 6. Monitoring and Alerts

[0373] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or use that violates the set policy, it generates an alert and notifies the user's device.

[0374] Hardware and software used

[0375] Hardware: Smartphones, servers

[0376] Software: Google's Natural Language API, Amazon Polly

[0377] Examples and prompts

[0378] Example user input: "My item arrived broken. What should I do?"

[0379] Prompt statement:

[0380] Please send the following text message to the Emotion Recognition API to analyze the emotion: My item arrived damaged. What should I do?

[0381] => Result: Anger

[0382] Please use the speech synthesis API to generate speech based on the following text message with the emotion "Anger." : Dear customer, we apologize for the inconvenience and will respond promptly.

[0383] => Audio data generation

[0384] Send the generated voice data to the customer

[0385] In this way, by using a specific person's voice data and adjusting the voice according to emotions based on text messages, more human-like communication becomes possible. In addition, rights management and fraudulent use monitoring functions ensure the rights of data providers are protected.

[0386] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0387] Step 1:

[0388] The user installs and launches a dedicated application on their smartphone. The user records voice data and enters metadata (such as name, age, gender, and a description of the voice). The device then sends the recorded voice data and the entered metadata to a server. The server analyzes the voice data, extracts features (such as pitch, tone, and sound quality), and stores them in a database.

[0389] Step 2:

[0390] The user logs in to the platform and sets the terms of use for the audio data. The terms of use set by the user (e.g., whether commercial use is permitted, the usage fee, etc.) are sent to the server via the terminal. The server updates the rights management information based on the received terms and conditions and stores it in a database.

[0391] Step 3:

[0392] The data user accesses the platform and creates a request for voice synthesis. The data user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device sends the input text message to the server, which calls the emotion recognition API to analyze the emotion. The server obtains the analyzed emotion and saves the results.

[0393] Step 4:

[0394] The server generates voice data using a speech synthesis AI model based on the text message and emotion recognition results. The generation AI model adjusts the intonation and nuance of the voice based on emotions (e.g., joy, sadness, anger) to create voice data. The generated voice data is stored on the server.

[0395] Step 5:

[0396] The server provides the generated voice data to the data user's device. The data user's device receives the voice data, and the user downloads and uses it. The server calculates profits based on the amount of voice data used and distributes them to the data provider.

[0397] Step 6:

[0398] The server continuously monitors the usage of voice data. To detect signs of fraudulent or unauthorized use, the monitoring system monitors for abnormal access or usage that violates established policies. If an abnormality is detected, the server generates an alert and notifies the data provider's device.

[0399] As described above, this system uses the voice data of a specific person to generate and provide voice that corresponds to the emotion of the text message, allowing users to communicate more humanly. It also manages the rights of the voice data and monitors unauthorized use, protecting the rights of the data provider.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0401] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0403] [Second embodiment]

[0404] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0405] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0406] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0408] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0410] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0411] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0412] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0415] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0416] The present invention is a system for providing a speech synthesis service based on the speech data of a specific person, and includes elements of a user, a terminal, and a server. Below, we will explain how the elements of this system work together to realize specific functions.

[0417] 1. Registering audio data

[0418] The user (voice data provider) installs a dedicated app on their device to record their voice. The user launches the app and records the voice to be provided. They also enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality).

[0419] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data and stores it in a database. It extracts features of the audio data (e.g., pitch, tone, and sound quality), generates identification information, and records it in the database.

[0420] 2. Rights Management

[0421] Users log in to the platform and set the terms of use for their own voice data (whether commercial use is permitted, usage fees, etc.). The device displays a form for entering this setting information and sends the user's input to the server.

[0422] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[0423] 3. Accepting synthesis requests

[0424] A user (data user) makes a request through the platform to generate audio based on a specific person's voice. The device provides a form for the user to select the provider of the audio data they want to use and to input the content they want to generate (such as a text message or lyrics). The user enters the request details, and the device sends this information to the server.

[0425] 4. Speech Synthesis

[0426] The server retrieves the corresponding voice data from the database based on the received synthesis request. Using a voice synthesis AI model, it inputs the text provided by the user and generates a realistic voice in the voice of a specific person. The synthesized voice data is stored in the database on the server.

[0427] 5. Provision of audio data

[0428] The server provides the generated voice data to the data user's terminal. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[0429] 6. Monitoring and Alerts

[0430] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[0431] Specific examples

[0432] For example, voice data provider A records his or her voice and registers it on the platform. A sets conditions allowing only commercial use. Meanwhile, advertising production company B wants to create a narration for a product introduction using provider A's voice and sends a request. The synthetic AI generates a narration in provider A's voice saying, "Please try this product," and provides it to B. The generated voice is for commercial use, and the compensation is distributed to provider A. During this process, the server monitors all usage and checks for any unauthorized use.

[0433] This system protects the rights of voice data providers and ensures appropriate profit distribution. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to users.

[0434] The processing flow will be explained below.

[0435] Step 1:

[0436] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[0437] Step 2:

[0438] The user presses the record button to record their own voice. After the recording is complete, the device temporarily stores the recorded voice data and displays a form for inputting necessary metadata (such as name, age, gender, and voice description).

[0439] Step 3:

[0440] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[0441] Step 4:

[0442] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[0443] Step 5:

[0444] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[0445] Step 6:

[0446] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[0447] Step 7:

[0448] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[0449] Step 8:

[0450] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[0451] Step 9:

[0452] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[0453] Step 10:

[0454] The server receives the request, retrieves the corresponding voice data from a database, and then uses a speech synthesis AI model to input the specified text and generate speech in the voice of a specific person.

[0455] Step 11:

[0456] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[0457] Step 12:

[0458] The server calculates profits based on the amount of voice data used, and the calculated profits are distributed to the data providers and recorded in a database.

[0459] Step 13:

[0460] The server continuously monitors the usage of voice data to detect signs of unauthorized use and generates an alert if an abnormality is detected.

[0461] Step 14:

[0462] The server sends the generated alert to the data provider's terminal, which displays the alert notification so that the user can check the status.

[0463] Through the above steps, this system enables the appropriate management and utilization of voice data, protecting the rights of data providers and providing new value.

[0464] Example 1

[0465] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0466] In speech synthesis systems based on voice data, there is a need to efficiently and safely manage the voice data of specific individuals, prevent unauthorized use, and realize accurate and realistic voice synthesis. In addition, a mechanism is needed to protect the rights of voice data providers and distribute profits appropriately.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0468] In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing rights to use the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a provision means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, a means for encrypting and transmitting metadata associated with the recorded voice data, and a prompt sentence generation means for inputting the voice data and text into a voice synthesis model. This enables safe management and use of voice data, protection of the rights of voice data providers, and realization of realistic voice synthesis.

[0469] The "registration means" is a mechanism for recording the voice data of a specific person and registering it in a database together with necessary metadata.

[0470] The "rights management means" is a mechanism for managing the rights to use audio data, and for setting and updating the conditions of use and permission details.

[0471] The "request receiving means" is a mechanism for receiving a request for voice synthesis based on voice data and transmitting the content of the request to the server.

[0472] The "voice synthesis means" is a mechanism for generating voice using voice data of a specific person based on a synthesis request.

[0473] The "means for providing" is a mechanism for providing the generated voice data to the terminal of the data user.

[0474] "Monitoring measures" are mechanisms for continuously monitoring the usage of voice data and detecting fraudulent or unauthorized use.

[0475] The "detection means" is a mechanism for determining whether or not there is unauthorized use of audio data based on information from the monitoring means.

[0476] The "notification means" is a mechanism for notifying relevant parties of an alert when unauthorized use of audio data is detected.

[0477] "Means for encryption and transmission" refers to a mechanism for encrypting recorded audio data and associated metadata and transmitting them to a server using a secure communication protocol.

[0478] The "prompt sentence generation means" is a mechanism for generating a prompt sentence for inputting voice data and text into a voice synthesis model.

[0479] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and includes elements of a user, a terminal, and a server. This system has the necessary configuration to realize the functions of voice data registration, rights management, reception of synthesis requests, voice synthesis, voice data provision, monitoring, and alerts.

[0480] Users install a dedicated app on their device to record their own voice. Using this app, users record their voice and enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality). The device encrypts the recorded voice data and the entered metadata and sends them to the server using a security protocol such as TLS (Transport Layer Security). The server analyzes the received voice data using acoustic analysis software (e.g., Praat or Audacity), extracts voice features, and stores them in a database.

[0481] Next, the user logs in to the platform and sets the conditions for using the audio data (whether commercial use is permitted, usage fees, etc.). The terminal displays a form for entering this setting information and sends the user's input to the server. The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions.

[0482] When generating audio using audio data, the user (data user) selects an audio data provider through the platform and inputs the content they want to generate (such as a text message or lyrics). For example, they input the text "Please try this product." The device then sends this information to the server.

[0483] The server retrieves the corresponding voice data from the database and inputs the text provided by the user using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). Specifically, the prompt "Try this product" is input into the speech synthesis model. The server stores the generated voice data in the database and then provides it to the user's device. The user can download the provided voice data and use it for the specified purpose.

[0484] Furthermore, the server continuously monitors the usage of voice data to detect signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or behavior that violates usage policies, it generates an alert and notifies the data provider's device, allowing the user to take appropriate action.

[0485] For example, if voice data provider A records his or her voice and registers on the platform under the condition that only commercial use is permitted, and an advertising production company wishes to create a narration in A's voice saying, "Please try this product." The server that receives this request through the platform uses A's voice to generate a narration using a voice synthesis model and provides it to the advertising production company. The generated voice is used commercially, and the compensation is distributed to A.

[0486] This system protects the rights of voice data providers and ensures appropriate profit sharing. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to each user.

[0487] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0488] Step 1: Record audio and enter metadata

[0489] The user installs a dedicated recording app on their device and launches it. The user presses the record button to record their own voice. Once recording is complete, the app displays a form in which the user can enter profile information (name, age, gender, voice quality, etc.). The data entered by the user here (audio data and metadata) is the input. The recorded audio data and the entered metadata are generated as the output.

[0490] Step 2: Encrypt and transmit audio data and metadata

[0491] The device encrypts the recorded audio data and input metadata using a security protocol such as TLS (Transport Layer Security). The encrypted audio data and metadata are input, and the output is sent to a server via a secure communication protocol.

[0492] Step 3: Analyze and save the audio data

[0493] The server decrypts the received encrypted audio data and metadata and analyzes it using acoustic analysis software (e.g., Praat or Audacity). This analysis extracts audio features (pitch, tone, timbre, etc.). The input is the decrypted audio data, and the output is the analyzed audio features and a process to store them in a database.

[0494] Step 4: Set the terms of use

[0495] The user logs in to a web or mobile platform and sets the terms of use for the audio data (whether commercial use is permitted, the usage fee, etc.). The terminal displays a form for inputting this setting information and accepts the user's input. The set terms of use are the input, and the terminal sends this information to the server. As an output, the rights management information in the database is updated.

[0496] Step 5: Accepting a synthesis request

[0497] The user (data user) selects a voice data provider and inputs the text of the content they want to generate. The input is the input text (e.g., a prompt sentence such as "Try this product"). The terminal sends this information to the server. The server receives a synthesis request as output.

[0498] Step 6: Acquiring audio data and synthesizing speech

[0499] The server retrieves the corresponding voice data from the database and inputs the text using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). The inputs are the retrieved voice data and the prompt. The speech synthesis AI model uses these to generate realistic speech. The output is the synthesized voice data, which is stored in the database.

[0500] Step 7: Provide synthetic voice data

[0501] The server provides the generated voice data to the user's (data user's) terminal. The provided voice data is the output. The user downloads this voice data and uses it for the specified purpose.

[0502] Step 8: Monitor usage and generate alerts

[0503] The server monitors the usage of voice data in real time. The input is the database access log. If the monitoring detects any fraudulent or unauthorized use, an alert is generated. The generated alert (notification message) is the output.

[0504] Step 9: Alert Notification

[0505] The server notifies the generated alert to the user's (data provider's) device. The alert notification (for example, app notification or email) is the output, allowing the user to take action against fraudulent use.

[0506] Through this series of steps, safe management and use of voice data, protection of the rights of voice data providers, and realistic voice synthesis are realized.

[0507] (Application example 1)

[0508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0509] In conventional virtual stores, when users ask questions about products or services in real time, only text-based responses are provided, which poses a problem of lacking realism and reliability. Furthermore, even when using speech synthesis technology, there is a lack of a mechanism for reproducing a specific person's voice in real time, which limits the interactive experience. For this reason, there is a need for a system that can respond in real time using a specific person's voice.

[0510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0511] In this invention, the server includes: a registration means for registering voice data of a specific person; a rights management means for managing usage rights of the voice data; a request receiving means for receiving a synthesis request; a voice synthesis means for synthesizing voice data based on the synthesis request; a provision means for providing the synthesized voice data; a monitoring means for monitoring the usage status of the voice data; a detection means for detecting unauthorized use of the voice data; a notification means for issuing an alert when unauthorized use is detected; a generation AI model utilization means for using a generation AI model when a user uses the voice synthesis service and synthesizing voice data in real time based on a prompt sentence; and a voice data provision means for providing the synthesized voice data to a terminal and receiving the voice data as a voice file, thereby enabling a user to receive explanations about products and services in real time in a virtual store from the voice of a specific person.

[0512] "Voice data of a specific person" is data that is a digital recording of a voice provided by a specific individual.

[0513] The "registration means" is a function that allows voice data providers to register their own voice data in the system.

[0514] The "rights management means" is a function that manages the conditions of use and rights information for audio data, and protects the rights of data providers.

[0515] The "request receiving means" is a function that allows a user of voice data to request voice synthesis using the voice of a specific person.

[0516] The "voice synthesis means" is a function for converting designated text into voice by using voice data of a specific person based on a request.

[0517] The "provision means" is a function for providing the generated voice data to the user.

[0518] "Monitoring means" is a function that monitors the usage of voice data and checks for any fraudulent or unauthorized use.

[0519] "Detection means" is a function for detecting unauthorized or improper use.

[0520] The "notification means" is a function for notifying the data provider of an alert when unauthorized use is detected.

[0521] "Means for utilizing a generative AI model" refers to a function for synthesizing voice data in real time based on a prompt sentence using a generative AI model.

[0522] The "voice data providing means" is a function that provides synthesized voice data to the terminal and allows the user to receive the voice data through a voice file.

[0523] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and is composed of a user, a terminal, and a server. This system includes functions for registering the voice data of a specific person, managing rights, accepting synthesis requests, voice synthesis, providing voice data, monitoring, detecting and notifying unauthorized use, utilizing a generative AI model, and providing data.

[0524] 1. Registering audio data

[0525] Users install an application on their device to record their own voice. They use this application to record voice data and enter metadata such as profile information and sound quality. The device then sends the recorded voice data and metadata to a server. The server analyzes the voice data and stores it in a database.

[0526] 2. Rights Management

[0527] The user logs in to the server and sets the conditions for using their own audio data. The terminal sends this information to the server. The server updates the rights management information in the database based on the condition information and generates a usage policy.

[0528] 3. Accepting synthesis requests

[0529] A user of voice data makes a request through a terminal to generate voice data based on the voice of a specific person. The terminal then selects the voice data provider they wish to use and provides a form for inputting the text message or question they wish to generate. This information is then sent to the server.

[0530] 4. Speech Synthesis

[0531] The server retrieves the corresponding voice data from the database based on the received synthesis request, and uses the generative AI model to synthesize a realistic voice with the voice of a specific person based on the prompt sentence. The generated voice data is stored in the database on the server.

[0532] 5. Provision of audio data

[0533] The server provides the generated voice data to the terminal, which the user can download and use for the specified purpose.

[0534] 6. Monitoring and Alerts

[0535] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. If unauthorized use is detected, the server generates an alert and notifies the data provider's device.

[0536] Specific examples

[0537] For example, if a user asks, "Please tell me about the ingredients in this new cosmetic product," the server uses a generative AI model to generate a response text. Based on this text, the server uses a speech synthesis API such as AWS Polly to generate voice data and respond in the voice of a specific person. This voice data is then provided to the user's device.

[0538] Example prompt sentence:

[0539] "Please tell me about the ingredients in this new cosmetic product."

[0540] In this way, by using this system, users can enjoy a realistic voice interaction experience and receive explanations about products and services in the virtual store.

[0541] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0542] Step 1:

[0543] Users use a dedicated application installed on their device to record their own voice and input metadata such as profile information and sound quality. Input: Voice data, profile information. Output: Recorded voice data, metadata.

[0544] Step 2:

[0545] The device sends the recorded audio data and input metadata to the server. Input: Recorded audio data, metadata. Output: Audio data and metadata sent to the server.

[0546] Step 3:

[0547] The server analyzes the received voice data and stores it in a voice database. At the same time, it extracts features from the voice data and records them in the database. Input: Voice data, metadata. Output: Stored voice data, features.

[0548] Step 4:

[0549] The user logs in to the server and sets the conditions for using their own audio data on the rights management screen. Input: Conditions of use (whether commercial use is permitted, usage fees, etc.). Output: The set conditions of use.

[0550] Step 5:

[0551] The terminal transmits the set usage conditions to the server, which records them in the rights management database. Input: Usage conditions. Output: Usage conditions recorded in the rights management database.

[0552] Step 6:

[0553] The user of the voice data makes a request through their terminal to generate voice based on the voice of a specific person. In the form, they enter the voice data provider they want to use and the text message they want to generate. Input: Voice data provider, text message. Output: Request details.

[0554] Step 7:

[0555] The terminal sends the request content to the server, and the server retrieves the corresponding voice data from the database. Input: Request content. Output: Retrieved voice data.

[0556] Step 8:

[0557] The server uses a generative AI model to convert text to speech in a specific person's voice based on the prompt and the acquired voice data. Input: Prompt, voice data. Output: Generated voice data.

[0558] Step 9:

[0559] The server stores the generated voice data in a database and provides it to the user's device. Input: Generated voice data. Output: Provided voice data.

[0560] Step 10:

[0561] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. Input: Usage data. Output: Monitoring reports, alerts.

[0562] Step 11:

[0563] If unauthorized use is detected, the server generates an alert and notifies the data provider's terminal. Input: Detection result. Output: Alert notification.

[0564] Step 12:

[0565] The user receives the voice data generated in real time through the terminal and uses it as an audio file. For example, if the question is "Please tell me about the ingredients of this new cosmetic product," the user will receive the answer in real time in a specific voice. Input: Provided voice data. Output: Playback of the audio file.

[0566] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0567] The present invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person, and its purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we will explain how the elements of this system work together to realize specific functions.

[0568] 1. Registering audio data

[0569] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input. The user presses the record button to record their own voice and enters the necessary metadata (name, age, gender, voice description, etc.).

[0570] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data, extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[0571] 2. Rights Management

[0572] The user logs in to the platform and sets the terms of use for their own voice data. The terminal displays a form that prompts the user to enter the terms of use (for example, whether commercial use is permitted, the usage fee, etc.). When the user enters the terms of use and presses the send button, the terminal sends the entered terms of use to the server.

[0573] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[0574] 3. Synthesis request acceptance and emotion recognition

[0575] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal provides a form in which the user selects a provider of the voice data they wish to use and inputs the content they wish to generate (e.g., text message or lyrics). The terminal then activates an emotion recognition means to recognize emotions from the user's input data and analyzes the emotions from the input text.

[0576] 4. Speech synthesis and emotion-based adjustment

[0577] The emotion recognition means analyzes emotions (e.g., joy, sadness, anger, etc.) from the input text and sends the emotion to the server, which retrieves the corresponding voice data from the database and generates voice using a voice synthesis AI model based on the user's emotion.

[0578] For example, if the input text is "I'm so happy today!", the emotion recognition means will determine this as "joy" and provide that information to the AI ​​model, which will then generate speech with intonation and nuances that match the emotion of joy.

[0579] 5. Provision of audio data

[0580] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[0581] 6. Monitoring and Alerts

[0582] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[0583] Specific examples

[0584] For example, voice data provider A records his or her voice and registers it on the platform. A sets a condition that only commercial use is permitted. Meanwhile, advertising production company B wants to create a narration for a product introduction in provider A's voice and sends a request. If the text provided by user B is "Please try this product," the emotion recognition means analyzes the text and recognizes a friendly emotion. Based on this emotional information, the server generates a narration in provider A's voice with a friendly tone. The generated voice is provided to advertising production company B, and the compensation is distributed to provider A.

[0585] This system protects the rights of voice data providers and enables high-quality voice synthesis that reflects emotions, thereby providing new value.

[0586] The processing flow will be explained below.

[0587] Step 1:

[0588] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[0589] Step 2:

[0590] The user presses the record button to record their own voice. Once the recording is complete, the device temporarily stores the recorded voice data and displays a form for entering necessary metadata (such as name, age, gender, and voice description).

[0591] Step 3:

[0592] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[0593] Step 4:

[0594] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[0595] Step 5:

[0596] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[0597] Step 6:

[0598] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[0599] Step 7:

[0600] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[0601] Step 8:

[0602] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[0603] Step 9:

[0604] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[0605] Step 10:

[0606] The server receives the request and retrieves the corresponding voice data from the database. Then, it uses an emotion recognition unit to analyze emotions based on the request text. The emotion recognition unit analyzes keywords and context within the text to identify specific emotions (e.g., joy, sadness, anger, etc.).

[0607] Step 11:

[0608] The server uses a speech synthesis AI model based on the analyzed emotional information to generate an appropriate voice from the voice data of a specific person. For example, if the text says "I'm so happy today!", the emotion recognition means will identify "joy" and the speech synthesis AI will generate a voice that expresses "joy."

[0609] Step 12:

[0610] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[0611] Step 13:

[0612] The server calculates profits based on the amount of voice data used, and records the calculated profits to be distributed to users (data providers).

[0613] Step 14:

[0614] The server continuously monitors voice data usage to detect signs of abuse and generates an alert if unauthorized use is detected.

[0615] Step 15:

[0616] The server sends the generated alert to the user's (data provider's) terminal, which displays the alert notification so that the user can check the status.

[0617] By combining emotion recognition and speech synthesis technologies, this system can generate and provide high-quality audio content that matches the user's emotions, thereby protecting the rights of audio data providers.

[0618] Example 2

[0619] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] Conventional speech synthesis systems have difficulty synthesizing a specific person's voice naturally, and speech synthesis based on emotion recognition in particular has not been fully realized. Furthermore, managing the rights to use voice data and monitoring and detecting unauthorized use are difficult, and the rights of data providers are not adequately protected. Therefore, there is a need for a system that not only provides high-quality speech synthesis but also reliably protects the rights of data providers.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0622] In this invention, the server includes a registration means for registering voice data of a specific person through a dedicated application installed on the user's device, an analysis means for analyzing the voice data and extracting features, a rights management means for allowing the user to set usage conditions for the voice data, a voice synthesis means for analyzing emotions from input text using an emotion recognition means and synthesizing voice based on the emotions, a provision means for storing the synthesized voice data in a database and providing it, a monitoring means for continuously monitoring the usage status of the voice data, and a notification means for detecting unauthorized use of the voice data and notifying the user. This not only enables high-quality voice synthesis and natural-sounding voice generation based on the user's emotions, but also protects the rights of data providers and makes it easy to monitor and detect unauthorized use.

[0623] A "dedicated app" is software with specific functions that is installed on a user's device and used.

[0624] "Voice data" refers to acoustic information that is a recording of a specific person's voice and stored in digital format.

[0625] "Features" are characteristic values ​​such as pitch, tone, and sound quality extracted from audio data.

[0626] "Rights management" refers to the procedures and systems for setting and managing the terms of use and rights information for audio data.

[0627] An "emotion recognition means" is a technology or tool that analyzes and identifies a user's emotions from input text or voice.

[0628] "Speech synthesis means" refers to technology and devices that generate new voices based on specific voice data and emotional information.

[0629] "Providing means" refers to the process and system for providing synthesized voice data to the user.

[0630] "Monitoring Measures" are technologies and tools that continuously monitor the usage of voice data.

[0631] The "notification means" refers to a system and method for notifying the user when unauthorized use or abnormality of audio data is detected.

[0632] The "profit distribution means" is a method and system for calculating profits according to the amount of use of voice data and distributing them to data providers.

[0633] A "voice synthesis AI model" is a model that uses artificial intelligence to generate high-quality voice based on voice data and emotional information.

[0634] This invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person. Its particular purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we explain how the elements of this system work together to realize specific functions.

[0635] 1. Registering audio data

[0636] The user installs a dedicated app on their device and launches it. The app has an interface for recording audio, and the user presses the record button to begin recording. When recording is complete, the user enters metadata such as their name, age, gender, and a description of the audio. The device then sends the recorded audio data and the entered metadata to a server. The server then analyzes the received audio data using audio analysis software (e.g., Praat, LibROSA), extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[0637] Examples:

[0638] When the user presses the record button, the device starts recording.

[0639] After recording, the user enters the metadata and sends it to the server.

[0640] The server analyzes the voice data and stores the features in a database.

[0641] 2. Rights Management

[0642] The user logs in to the platform and sets the terms of use for the audio data. The device displays a form for entering the terms of use (e.g., whether commercial use is permitted, and the usage fee). The user enters the terms of use, and the device sends the information to the server. The server updates the rights management information in the database based on the received terms and generates a usage policy based on the terms of use.

[0643] Examples:

[0644] The user logs into the platform and enters the terms of use.

[0645] The terminal transmits the entered usage conditions to the server.

[0646] The server generates a usage policy and stores it in a database.

[0647] 3. Synthesis request acceptance and emotion recognition

[0648] The user (data user) accesses the platform and creates a request for speech synthesis. The device provides a form where the user can select the provider of the voice data they want to use and input the content they want to generate (e.g., text message or lyrics). The device then sends the input text to emotion recognition software (e.g., Google Cloud Natural Language, IBM Watson), which analyzes emotions from the text.

[0649] Examples:

[0650] The user inputs the content they want to generate.

[0651] The device sends the input text to emotion recognition software, which analyzes the emotion.

[0652] 4. Speech synthesis and emotion-based adjustment

[0653] The emotion recognition software analyzes the emotional information and sends it to the server, which retrieves the corresponding voice data from the database and generates speech using a speech synthesis AI model (e.g., Tacotron, WaveNet). The generated speech is adjusted according to the emotion.

[0654] Examples:

[0655] The server acquires the emotion information and acquires the voice data from the database.

[0656] The server uses a speech synthesis AI model to generate speech that corresponds to the emotion.

[0657] 5. Provision of audio data

[0658] The server stores the generated voice data in a database and provides it to the data user's device. The data user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[0659] Examples:

[0660] The server stores the generated audio data in a database.

[0661] The device downloads the audio data.

[0662] Example prompt sentence:

[0663] "Using the voice of voice data provider A, synthesize the following text with a friendly emotion: 'Try this product.'"

[0664] 6. Monitoring and Alerts

[0665] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[0666] Examples:

[0667] The server monitors the access log and detects any abnormalities.

[0668] The server sends an alert notification to the user.

[0669] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0670] Step 1:

[0671] The user installs the dedicated app on their device and launches it. When the user presses the record button, the device starts recording the audio. After recording is complete, the device accepts metadata entered by the user (such as name, age, gender, and audio description).

[0672] Input: User starts recording and enters metadata

[0673] Output: Recorded audio data and metadata

[0674] Specific behavior:

[0675] The user presses the record button.

[0676] The device will record and temporarily store the audio.

[0677] The user enters the required metadata.

[0678] Step 2:

[0679] The device sends the recorded voice data and input metadata to a server, which then analyzes the received voice data using voice analysis software, extracts features (pitch, tone, timbre, etc.), and stores them in a database.

[0680] Input: Audio data and metadata sent from the device

[0681] Output: Analyzed feature data

[0682] Specific behavior:

[0683] The device encrypts the audio data and metadata and sends them to the server.

[0684] The server reads the voice data and extracts features using analysis software.

[0685] The feature data is stored in a database.

[0686] Step 3:

[0687] The user logs in to the platform and sets the terms of use for the audio data. They enter the terms of use (whether commercial use is permitted, the usage fee, etc.) in a form displayed on the device, and the device sends this information to the server. The server updates the rights management information in the database based on the received terms and conditions and generates a usage policy.

[0688] Input: Terms of Use Information

[0689] Output: Updated rights management information and usage policy

[0690] Specific behavior:

[0691] The user logs into the platform and enters the terms of use.

[0692] The terminal transmits the usage conditions to the server.

[0693] The server updates the database based on the condition information and generates a usage policy.

[0694] Step 4:

[0695] The user (data user) accesses the platform and creates a request for voice synthesis. The device provides a form where the user can select the provider of the voice data they want to use and input the content they want to generate (text message or lyrics). The device sends the input text to emotion recognition software, which analyzes emotions from the text.

[0696] Input: The content (text) you want to generate

[0697] Output: Parsed emotion information

[0698] Specific behavior:

[0699] The user selects a provider and enters text content.

[0700] The device sends the text to emotion recognition software.

[0701] Emotion recognition software analyzes emotions from text and returns emotional information.

[0702] Step 5:

[0703] The emotion recognition software analyzes the emotional information and sends it to a server, which retrieves the corresponding voice data from a database and generates a voice using a voice synthesis AI model. The generated voice is adjusted according to the emotion.

[0704] Input: Emotional information, voice data

[0705] Output: Generated audio data

[0706] Specific behavior:

[0707] The server retrieves the audio data from the database.

[0708] The server inputs emotional information and voice data into the AI ​​model.

[0709] The AI ​​model generates voice based on the emotion.

[0710] Step 6:

[0711] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[0712] Input: Generated audio data

[0713] Output: Provided audio data, profits distributed

[0714] Specific behavior:

[0715] The server stores the generated audio data in a database.

[0716] The server sends a download link for the audio data to the device.

[0717] The device downloads the audio data.

[0718] The server calculates profits based on usage and distributes them to data providers.

[0719] Step 7:

[0720] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[0721] Input: Voice data usage

[0722] Output: Alert notification

[0723] Specific behavior:

[0724] The server monitors the access log and detects any abnormalities.

[0725] The server generates an alert and notifies the user.

[0726] (Application example 2)

[0727] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0728] While conventional speech synthesis technologies can generate voice data using a specific person's voice, they cannot automatically adjust the voice according to emotion, making it difficult to communicate naturally with users. Furthermore, there is a lack of effective systems for managing voice data usage rights and detecting unauthorized use. There is a need for a system that can recognize emotions and adjust the voice according to the user's text message in real time.

[0729] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing usage rights of the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a providing means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, an emotion recognition means for analyzing emotions from an input text message, and a voice synthesis adjustment means for adjusting the voice based on the analyzed emotions. This makes it possible to use the voice data of a specific person and adjust the voice based on emotions corresponding to the text message, thereby realizing more human-like communication.

[0730] The "registration means" is a function for registering voice data of a specific person in the system.

[0731] The "rights management means" is a function for appropriately managing the rights to use audio data and protecting the rights of data providers.

[0732] The "request receiving means" is a function for receiving a request to synthesize voice data.

[0733] The "voice synthesis means" is a function for generating voice data based on a synthesis request.

[0734] The "provision means" is a function for providing synthesized voice data to the user.

[0735] The "monitoring means" is a function for continuously monitoring the usage status of voice data.

[0736] The "detection means" is a function for detecting unauthorized use of audio data.

[0737] The "notification means" is a function for sending an alert to the user when unauthorized use is detected.

[0738] The "emotion recognition means" is a function for analyzing emotions from a text message entered by a user.

[0739] The "voice synthesis adjustment means" is a function for adjusting the voice data to be generated based on the analyzed emotion.

[0740] A specific system configuration and its operation will be described below for an embodiment of the present invention.

[0741] 1. Registering audio data

[0742] The user installs a dedicated application on their smartphone. They launch the application and register voice data through an interface that allows them to record their own voice. The user presses the record button, records their own voice, and enters the necessary metadata (such as name, age, gender, and voice description). The device then sends the recorded voice data and metadata to the server. The server analyzes the voice data, extracts features, and stores them in a database.

[0743] 2. Rights Management

[0744] Users log in to the platform and set the terms of use for the audio data (for example, whether commercial use is permitted, and the usage fee setting). The setting information is sent to the server, which then updates the rights management information and generates a usage policy based on that information.

[0745] 3. Synthesis request acceptance and emotion recognition

[0746] A user (data user) accesses the platform and creates a request for voice synthesis. The user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device then activates the emotion recognition means and analyzes emotions from the input text.

[0747] 4. Speech synthesis and emotion-based adjustment

[0748] The emotion recognition means analyzes the text message and determines the emotion (e.g., joy, sadness, anger, etc.). This emotion information is sent to the server, which retrieves the corresponding voice data from a database. A speech synthesis AI model is used to generate a voice corresponding to the emotion. For example, for the input text "I'm so happy today!", the system detects the emotion "joy" and generates a voice with intonation and nuance that matches the emotion of joy.

[0749] 5. Provision of audio data

[0750] The server stores the generated voice data in a database and provides it to the user's device. The user downloads the provided voice data and uses it for the specified purpose.

[0751] 6. Monitoring and Alerts

[0752] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or use that violates the set policy, it generates an alert and notifies the user's device.

[0753] Hardware and software used

[0754] Hardware: Smartphones, servers

[0755] Software: Google's Natural Language API, Amazon Polly

[0756] Examples and prompts

[0757] Example user input: "My item arrived broken. What should I do?"

[0758] Prompt statement:

[0759] Please send the following text message to the Emotion Recognition API to analyze the emotion: My item arrived damaged. What should I do?

[0760] => Result: Anger

[0761] Please use the speech synthesis API to generate speech based on the following text message with the emotion "Anger." : Dear customer, we apologize for the inconvenience and will respond promptly.

[0762] => Audio data generation

[0763] Send the generated voice data to the customer

[0764] In this way, by using a specific person's voice data and adjusting the voice according to emotions based on text messages, more human-like communication becomes possible. In addition, rights management and fraudulent use monitoring functions ensure the rights of data providers are protected.

[0765] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0766] Step 1:

[0767] The user installs and launches a dedicated application on their smartphone. The user records voice data and enters metadata (such as name, age, gender, and a description of the voice). The device then sends the recorded voice data and the entered metadata to a server. The server analyzes the voice data, extracts features (such as pitch, tone, and sound quality), and stores them in a database.

[0768] Step 2:

[0769] The user logs in to the platform and sets the terms of use for the audio data. The terms of use set by the user (e.g., whether commercial use is permitted, the usage fee, etc.) are sent to the server via the terminal. The server updates the rights management information based on the received terms and conditions and stores it in a database.

[0770] Step 3:

[0771] The data user accesses the platform and creates a request for voice synthesis. The data user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device sends the input text message to the server, which calls the emotion recognition API to analyze the emotion. The server obtains the analyzed emotion and saves the results.

[0772] Step 4:

[0773] The server generates voice data using a speech synthesis AI model based on the text message and emotion recognition results. The generation AI model adjusts the intonation and nuance of the voice based on emotions (e.g., joy, sadness, anger) to create voice data. The generated voice data is stored on the server.

[0774] Step 5:

[0775] The server provides the generated voice data to the data user's device. The data user's device receives the voice data, and the user downloads and uses it. The server calculates profits based on the amount of voice data used and distributes them to the data provider.

[0776] Step 6:

[0777] The server continuously monitors the usage of voice data. To detect signs of fraudulent or unauthorized use, the monitoring system monitors for abnormal access or usage that violates established policies. If an abnormality is detected, the server generates an alert and notifies the data provider's device.

[0778] As described above, this system uses the voice data of a specific person to generate and provide voice that corresponds to the emotion of the text message, allowing users to communicate more humanly. It also manages the rights of the voice data and monitors unauthorized use, protecting the rights of the data provider.

[0779] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0780] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0781] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0782] [Third embodiment]

[0783] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0784] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0785] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0786] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0787] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0788] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0789] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0790] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0791] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0792] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0793] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0794] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0795] The present invention is a system for providing a speech synthesis service based on the speech data of a specific person, and includes elements of a user, a terminal, and a server. Below, we will explain how the elements of this system work together to realize specific functions.

[0796] 1. Registering audio data

[0797] The user (voice data provider) installs a dedicated app on their device to record their voice. The user launches the app and records the voice to be provided. They also enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality).

[0798] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data and stores it in a database. It extracts features of the audio data (e.g., pitch, tone, and sound quality), generates identification information, and records it in the database.

[0799] 2. Rights Management

[0800] Users log in to the platform and set the terms of use for their own voice data (whether commercial use is permitted, usage fees, etc.). The device displays a form for entering this setting information and sends the user's input to the server.

[0801] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[0802] 3. Accepting synthesis requests

[0803] A user (data user) makes a request through the platform to generate audio based on a specific person's voice. The device provides a form for the user to select the provider of the audio data they want to use and to input the content they want to generate (such as a text message or lyrics). The user enters the request details, and the device sends this information to the server.

[0804] 4. Speech Synthesis

[0805] The server retrieves the corresponding voice data from the database based on the received synthesis request. Using a voice synthesis AI model, it inputs the text provided by the user and generates a realistic voice in the voice of a specific person. The synthesized voice data is stored in the database on the server.

[0806] 5. Provision of audio data

[0807] The server provides the generated voice data to the data user's terminal. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[0808] 6. Monitoring and Alerts

[0809] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[0810] Specific examples

[0811] For example, voice data provider A records his or her voice and registers it on the platform. A sets conditions allowing only commercial use. Meanwhile, advertising production company B wants to create a narration for a product introduction using provider A's voice and sends a request. The synthetic AI generates a narration in provider A's voice saying, "Please try this product," and provides it to B. The generated voice is for commercial use, and the compensation is distributed to provider A. During this process, the server monitors all usage and checks for any unauthorized use.

[0812] This system protects the rights of voice data providers and ensures appropriate profit distribution. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to users.

[0813] The processing flow will be explained below.

[0814] Step 1:

[0815] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[0816] Step 2:

[0817] The user presses the record button to record their own voice. After the recording is complete, the device temporarily stores the recorded voice data and displays a form for inputting necessary metadata (such as name, age, gender, and voice description).

[0818] Step 3:

[0819] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[0820] Step 4:

[0821] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[0822] Step 5:

[0823] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[0824] Step 6:

[0825] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[0826] Step 7:

[0827] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[0828] Step 8:

[0829] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[0830] Step 9:

[0831] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[0832] Step 10:

[0833] The server receives the request, retrieves the corresponding voice data from a database, and then uses a speech synthesis AI model to input the specified text and generate speech in the voice of a specific person.

[0834] Step 11:

[0835] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[0836] Step 12:

[0837] The server calculates profits based on the amount of voice data used, and the calculated profits are distributed to the data providers and recorded in a database.

[0838] Step 13:

[0839] The server continuously monitors the usage of voice data to detect signs of unauthorized use and generates an alert if an abnormality is detected.

[0840] Step 14:

[0841] The server sends the generated alert to the data provider's terminal, which displays the alert notification so that the user can check the status.

[0842] Through the above steps, this system enables the appropriate management and utilization of voice data, protecting the rights of data providers and providing new value.

[0843] Example 1

[0844] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0845] In speech synthesis systems based on voice data, there is a need to efficiently and safely manage the voice data of specific individuals, prevent unauthorized use, and realize accurate and realistic voice synthesis. In addition, a mechanism is needed to protect the rights of voice data providers and distribute profits appropriately.

[0846] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0847] In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing rights to use the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a provision means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, a means for encrypting and transmitting metadata associated with the recorded voice data, and a prompt sentence generation means for inputting the voice data and text into a voice synthesis model. This enables safe management and use of voice data, protection of the rights of voice data providers, and realization of realistic voice synthesis.

[0848] The "registration means" is a mechanism for recording the voice data of a specific person and registering it in a database together with necessary metadata.

[0849] The "rights management means" is a mechanism for managing the rights to use audio data, and for setting and updating the conditions of use and permission details.

[0850] The "request receiving means" is a mechanism for receiving a request for voice synthesis based on voice data and transmitting the content of the request to the server.

[0851] The "voice synthesis means" is a mechanism for generating voice using voice data of a specific person based on a synthesis request.

[0852] The "means for providing" is a mechanism for providing the generated voice data to the terminal of the data user.

[0853] "Monitoring measures" are mechanisms for continuously monitoring the usage of voice data and detecting fraudulent or unauthorized use.

[0854] The "detection means" is a mechanism for determining whether or not there is unauthorized use of audio data based on information from the monitoring means.

[0855] The "notification means" is a mechanism for notifying relevant parties of an alert when unauthorized use of audio data is detected.

[0856] "Means for encryption and transmission" refers to a mechanism for encrypting recorded audio data and associated metadata and transmitting them to a server using a secure communication protocol.

[0857] The "prompt sentence generation means" is a mechanism for generating a prompt sentence for inputting voice data and text into a voice synthesis model.

[0858] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and includes elements of a user, a terminal, and a server. This system has the necessary configuration to realize the functions of voice data registration, rights management, reception of synthesis requests, voice synthesis, voice data provision, monitoring, and alerts.

[0859] Users install a dedicated app on their device to record their own voice. Using this app, users record their voice and enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality). The device encrypts the recorded voice data and the entered metadata and sends them to the server using a security protocol such as TLS (Transport Layer Security). The server analyzes the received voice data using acoustic analysis software (e.g., Praat or Audacity), extracts voice features, and stores them in a database.

[0860] Next, the user logs in to the platform and sets the conditions for using the audio data (whether commercial use is permitted, usage fees, etc.). The terminal displays a form for entering this setting information and sends the user's input to the server. The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions.

[0861] When generating audio using audio data, the user (data user) selects an audio data provider through the platform and inputs the content they want to generate (such as a text message or lyrics). For example, they input the text "Please try this product." The device then sends this information to the server.

[0862] The server retrieves the corresponding voice data from the database and inputs the text provided by the user using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). Specifically, the prompt "Try this product" is input into the speech synthesis model. The server stores the generated voice data in the database and then provides it to the user's device. The user can download the provided voice data and use it for the specified purpose.

[0863] Furthermore, the server continuously monitors the usage of voice data to detect signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or behavior that violates usage policies, it generates an alert and notifies the data provider's device, allowing the user to take appropriate action.

[0864] For example, if voice data provider A records his or her voice and registers on the platform under the condition that only commercial use is permitted, and an advertising production company wishes to create a narration in A's voice saying, "Please try this product." The server that receives this request through the platform uses A's voice to generate a narration using a voice synthesis model and provides it to the advertising production company. The generated voice is used commercially, and the compensation is distributed to A.

[0865] This system protects the rights of voice data providers and ensures appropriate profit sharing. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to each user.

[0866] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0867] Step 1: Record audio and enter metadata

[0868] The user installs a dedicated recording app on their device and launches it. The user presses the record button to record their own voice. Once recording is complete, the app displays a form in which the user can enter profile information (name, age, gender, voice quality, etc.). The data entered by the user here (audio data and metadata) is the input. The recorded audio data and the entered metadata are generated as the output.

[0869] Step 2: Encrypt and transmit audio data and metadata

[0870] The device encrypts the recorded audio data and input metadata using a security protocol such as TLS (Transport Layer Security). The encrypted audio data and metadata are input, and the output is sent to a server via a secure communication protocol.

[0871] Step 3: Analyze and save the audio data

[0872] The server decrypts the received encrypted audio data and metadata and analyzes it using acoustic analysis software (e.g., Praat or Audacity). This analysis extracts audio features (pitch, tone, timbre, etc.). The input is the decrypted audio data, and the output is the analyzed audio features and a process to store them in a database.

[0873] Step 4: Set the terms of use

[0874] The user logs in to a web or mobile platform and sets the terms of use for the audio data (whether commercial use is permitted, the usage fee, etc.). The terminal displays a form for inputting this setting information and accepts the user's input. The set terms of use are the input, and the terminal sends this information to the server. As an output, the rights management information in the database is updated.

[0875] Step 5: Accepting a synthesis request

[0876] The user (data user) selects a voice data provider and inputs the text of the content they want to generate. The input is the input text (e.g., a prompt sentence such as "Try this product"). The terminal sends this information to the server. The server receives a synthesis request as output.

[0877] Step 6: Acquiring audio data and synthesizing speech

[0878] The server retrieves the corresponding voice data from the database and inputs the text using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). The inputs are the retrieved voice data and the prompt. The speech synthesis AI model uses these to generate realistic speech. The output is the synthesized voice data, which is stored in the database.

[0879] Step 7: Provide synthetic voice data

[0880] The server provides the generated voice data to the user's (data user's) terminal. The provided voice data is the output. The user downloads this voice data and uses it for the specified purpose.

[0881] Step 8: Monitor usage and generate alerts

[0882] The server monitors the usage of voice data in real time. The input is the database access log. If the monitoring detects any fraudulent or unauthorized use, an alert is generated. The generated alert (notification message) is the output.

[0883] Step 9: Alert Notification

[0884] The server notifies the generated alert to the user's (data provider's) device. The alert notification (for example, app notification or email) is the output, allowing the user to take action against fraudulent use.

[0885] Through this series of steps, safe management and use of voice data, protection of the rights of voice data providers, and realistic voice synthesis are realized.

[0886] (Application example 1)

[0887] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0888] In conventional virtual stores, when users ask questions about products or services in real time, only text-based responses are provided, which poses a problem of lacking realism and reliability. Furthermore, even when using speech synthesis technology, there is a lack of a mechanism for reproducing a specific person's voice in real time, which limits the interactive experience. For this reason, there is a need for a system that can respond in real time using a specific person's voice.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0890] In this invention, the server includes: a registration means for registering voice data of a specific person; a rights management means for managing usage rights of the voice data; a request receiving means for receiving a synthesis request; a voice synthesis means for synthesizing voice data based on the synthesis request; a provision means for providing the synthesized voice data; a monitoring means for monitoring the usage status of the voice data; a detection means for detecting unauthorized use of the voice data; a notification means for issuing an alert when unauthorized use is detected; a generation AI model utilization means for using a generation AI model when a user uses the voice synthesis service and synthesizing voice data in real time based on a prompt sentence; and a voice data provision means for providing the synthesized voice data to a terminal and receiving the voice data as a voice file, thereby enabling a user to receive explanations about products and services in real time in a virtual store from the voice of a specific person.

[0891] "Voice data of a specific person" is data that is a digital recording of a voice provided by a specific individual.

[0892] The "registration means" is a function that allows voice data providers to register their own voice data in the system.

[0893] The "rights management means" is a function that manages the conditions of use and rights information for audio data, and protects the rights of data providers.

[0894] The "request receiving means" is a function that allows a user of voice data to request voice synthesis using the voice of a specific person.

[0895] The "voice synthesis means" is a function for converting designated text into voice by using voice data of a specific person based on a request.

[0896] The "provision means" is a function for providing the generated voice data to the user.

[0897] "Monitoring means" is a function that monitors the usage of voice data and checks for any fraudulent or unauthorized use.

[0898] "Detection means" is a function for detecting unauthorized or improper use.

[0899] The "notification means" is a function for notifying the data provider of an alert when unauthorized use is detected.

[0900] "Means for utilizing a generative AI model" refers to a function for synthesizing voice data in real time based on a prompt sentence using a generative AI model.

[0901] The "voice data providing means" is a function that provides synthesized voice data to the terminal and allows the user to receive the voice data through a voice file.

[0902] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and is composed of a user, a terminal, and a server. This system includes functions for registering the voice data of a specific person, managing rights, accepting synthesis requests, voice synthesis, providing voice data, monitoring, detecting and notifying unauthorized use, utilizing a generative AI model, and providing data.

[0903] 1. Registering audio data

[0904] Users install an application on their device to record their own voice. They use this application to record voice data and enter metadata such as profile information and sound quality. The device then sends the recorded voice data and metadata to a server. The server analyzes the voice data and stores it in a database.

[0905] 2. Rights Management

[0906] The user logs in to the server and sets the conditions for using their own audio data. The terminal sends this information to the server. The server updates the rights management information in the database based on the condition information and generates a usage policy.

[0907] 3. Accepting synthesis requests

[0908] A user of voice data makes a request through a terminal to generate voice data based on the voice of a specific person. The terminal then selects the voice data provider they wish to use and provides a form for inputting the text message or question they wish to generate. This information is then sent to the server.

[0909] 4. Speech Synthesis

[0910] The server retrieves the corresponding voice data from the database based on the received synthesis request, and uses the generative AI model to synthesize a realistic voice with the voice of a specific person based on the prompt sentence. The generated voice data is stored in the database on the server.

[0911] 5. Provision of audio data

[0912] The server provides the generated voice data to the terminal, which the user can download and use for the specified purpose.

[0913] 6. Monitoring and Alerts

[0914] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. If unauthorized use is detected, the server generates an alert and notifies the data provider's device.

[0915] Specific examples

[0916] For example, if a user asks, "Please tell me about the ingredients in this new cosmetic product," the server uses a generative AI model to generate a response text. Based on this text, the server uses a speech synthesis API such as AWS Polly to generate voice data and respond in the voice of a specific person. This voice data is then provided to the user's device.

[0917] Example prompt sentence:

[0918] "Please tell me about the ingredients in this new cosmetic product."

[0919] In this way, by using this system, users can enjoy a realistic voice interaction experience and receive explanations about products and services in the virtual store.

[0920] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0921] Step 1:

[0922] Users use a dedicated application installed on their device to record their own voice and input metadata such as profile information and sound quality. Input: Voice data, profile information. Output: Recorded voice data, metadata.

[0923] Step 2:

[0924] The device sends the recorded audio data and input metadata to the server. Input: Recorded audio data, metadata. Output: Audio data and metadata sent to the server.

[0925] Step 3:

[0926] The server analyzes the received voice data and stores it in a voice database. At the same time, it extracts features from the voice data and records them in the database. Input: Voice data, metadata. Output: Stored voice data, features.

[0927] Step 4:

[0928] The user logs in to the server and sets the conditions for using their own audio data on the rights management screen. Input: Conditions of use (whether commercial use is permitted, usage fees, etc.). Output: The set conditions of use.

[0929] Step 5:

[0930] The terminal transmits the set usage conditions to the server, which records them in the rights management database. Input: Usage conditions. Output: Usage conditions recorded in the rights management database.

[0931] Step 6:

[0932] The user of the voice data makes a request through their terminal to generate voice based on the voice of a specific person. In the form, they enter the voice data provider they want to use and the text message they want to generate. Input: Voice data provider, text message. Output: Request details.

[0933] Step 7:

[0934] The terminal sends the request content to the server, and the server retrieves the corresponding voice data from the database. Input: Request content. Output: Retrieved voice data.

[0935] Step 8:

[0936] The server uses a generative AI model to convert text to speech in a specific person's voice based on the prompt and the acquired voice data. Input: Prompt, voice data. Output: Generated voice data.

[0937] Step 9:

[0938] The server stores the generated voice data in a database and provides it to the user's device. Input: Generated voice data. Output: Provided voice data.

[0939] Step 10:

[0940] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. Input: Usage data. Output: Monitoring reports, alerts.

[0941] Step 11:

[0942] If unauthorized use is detected, the server generates an alert and notifies the data provider's terminal. Input: Detection result. Output: Alert notification.

[0943] Step 12:

[0944] The user receives the voice data generated in real time through the terminal and uses it as an audio file. For example, if the question is "Please tell me about the ingredients of this new cosmetic product," the user will receive the answer in real time in a specific voice. Input: Provided voice data. Output: Playback of the audio file.

[0945] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0946] The present invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person, and its purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we will explain how the elements of this system work together to realize specific functions.

[0947] 1. Registering audio data

[0948] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input. The user presses the record button to record their own voice and enters the necessary metadata (name, age, gender, voice description, etc.).

[0949] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data, extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[0950] 2. Rights Management

[0951] The user logs in to the platform and sets the terms of use for their own voice data. The terminal displays a form that prompts the user to enter the terms of use (for example, whether commercial use is permitted, the usage fee, etc.). When the user enters the terms of use and presses the send button, the terminal sends the entered terms of use to the server.

[0952] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[0953] 3. Synthesis request acceptance and emotion recognition

[0954] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal provides a form in which the user selects a provider of the voice data they wish to use and inputs the content they wish to generate (e.g., text message or lyrics). The terminal then activates an emotion recognition means to recognize emotions from the user's input data and analyzes the emotions from the input text.

[0955] 4. Speech synthesis and emotion-based adjustment

[0956] The emotion recognition means analyzes emotions (e.g., joy, sadness, anger, etc.) from the input text and sends the emotion to the server, which retrieves the corresponding voice data from the database and generates voice using a voice synthesis AI model based on the user's emotion.

[0957] For example, if the input text is "I'm so happy today!", the emotion recognition means will determine this as "joy" and provide that information to the AI ​​model, which will then generate speech with intonation and nuances that match the emotion of joy.

[0958] 5. Provision of audio data

[0959] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[0960] 6. Monitoring and Alerts

[0961] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[0962] Specific examples

[0963] For example, voice data provider A records his or her voice and registers it on the platform. A sets a condition that only commercial use is permitted. Meanwhile, advertising production company B wants to create a narration for a product introduction in provider A's voice and sends a request. If the text provided by user B is "Please try this product," the emotion recognition means analyzes the text and recognizes a friendly emotion. Based on this emotional information, the server generates a narration in provider A's voice with a friendly tone. The generated voice is provided to advertising production company B, and the compensation is distributed to provider A.

[0964] This system protects the rights of voice data providers and enables high-quality voice synthesis that reflects emotions, thereby providing new value.

[0965] The processing flow will be explained below.

[0966] Step 1:

[0967] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[0968] Step 2:

[0969] The user presses the record button to record their own voice. Once the recording is complete, the device temporarily stores the recorded voice data and displays a form for entering necessary metadata (such as name, age, gender, and voice description).

[0970] Step 3:

[0971] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[0972] Step 4:

[0973] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[0974] Step 5:

[0975] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[0976] Step 6:

[0977] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[0978] Step 7:

[0979] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[0980] Step 8:

[0981] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[0982] Step 9:

[0983] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[0984] Step 10:

[0985] The server receives the request and retrieves the corresponding voice data from the database. Then, it uses an emotion recognition unit to analyze emotions based on the request text. The emotion recognition unit analyzes keywords and context within the text to identify specific emotions (e.g., joy, sadness, anger, etc.).

[0986] Step 11:

[0987] The server uses a speech synthesis AI model based on the analyzed emotional information to generate an appropriate voice from the voice data of a specific person. For example, if the text says "I'm so happy today!", the emotion recognition means will identify "joy" and the speech synthesis AI will generate a voice that expresses "joy."

[0988] Step 12:

[0989] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[0990] Step 13:

[0991] The server calculates profits based on the amount of voice data used, and records the calculated profits to be distributed to users (data providers).

[0992] Step 14:

[0993] The server continuously monitors voice data usage to detect signs of abuse and generates an alert if unauthorized use is detected.

[0994] Step 15:

[0995] The server sends the generated alert to the user's (data provider's) terminal, which displays the alert notification so that the user can check the status.

[0996] By combining emotion recognition and speech synthesis technologies, this system can generate and provide high-quality audio content that matches the user's emotions, thereby protecting the rights of audio data providers.

[0997] Example 2

[0998] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0999] Conventional speech synthesis systems have difficulty synthesizing a specific person's voice naturally, and speech synthesis based on emotion recognition in particular has not been fully realized. Furthermore, managing the rights to use voice data and monitoring and detecting unauthorized use are difficult, and the rights of data providers are not adequately protected. Therefore, there is a need for a system that not only provides high-quality speech synthesis but also reliably protects the rights of data providers.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1001] In this invention, the server includes a registration means for registering voice data of a specific person through a dedicated application installed on the user's device, an analysis means for analyzing the voice data and extracting features, a rights management means for allowing the user to set usage conditions for the voice data, a voice synthesis means for analyzing emotions from input text using an emotion recognition means and synthesizing voice based on the emotions, a provision means for storing the synthesized voice data in a database and providing it, a monitoring means for continuously monitoring the usage status of the voice data, and a notification means for detecting unauthorized use of the voice data and notifying the user. This not only enables high-quality voice synthesis and natural-sounding voice generation based on the user's emotions, but also protects the rights of data providers and makes it easy to monitor and detect unauthorized use.

[1002] A "dedicated app" is software with specific functions that is installed on a user's device and used.

[1003] "Voice data" refers to acoustic information that is a recording of a specific person's voice and stored in digital format.

[1004] "Features" are characteristic values ​​such as pitch, tone, and sound quality extracted from audio data.

[1005] "Rights management" refers to the procedures and systems for setting and managing the terms of use and rights information for audio data.

[1006] An "emotion recognition means" is a technology or tool that analyzes and identifies a user's emotions from input text or voice.

[1007] "Speech synthesis means" refers to technology and devices that generate new voices based on specific voice data and emotional information.

[1008] "Providing means" refers to the process and system for providing synthesized voice data to the user.

[1009] "Monitoring Measures" are technologies and tools that continuously monitor the usage of voice data.

[1010] The "notification means" refers to a system and method for notifying the user when unauthorized use or abnormality of audio data is detected.

[1011] The "profit distribution means" is a method and system for calculating profits according to the amount of use of voice data and distributing them to data providers.

[1012] A "voice synthesis AI model" is a model that uses artificial intelligence to generate high-quality voice based on voice data and emotional information.

[1013] This invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person. Its particular purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we explain how the elements of this system work together to realize specific functions.

[1014] 1. Registering audio data

[1015] The user installs a dedicated app on their device and launches it. The app has an interface for recording audio, and the user presses the record button to begin recording. When recording is complete, the user enters metadata such as their name, age, gender, and a description of the audio. The device then sends the recorded audio data and the entered metadata to a server. The server then analyzes the received audio data using audio analysis software (e.g., Praat, LibROSA), extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[1016] Examples:

[1017] When the user presses the record button, the device starts recording.

[1018] After recording, the user enters the metadata and sends it to the server.

[1019] The server analyzes the voice data and stores the features in a database.

[1020] 2. Rights Management

[1021] The user logs in to the platform and sets the terms of use for the audio data. The device displays a form for entering the terms of use (e.g., whether commercial use is permitted, and the usage fee). The user enters the terms of use, and the device sends the information to the server. The server updates the rights management information in the database based on the received terms and generates a usage policy based on the terms of use.

[1022] Examples:

[1023] The user logs into the platform and enters the terms of use.

[1024] The terminal transmits the entered usage conditions to the server.

[1025] The server generates a usage policy and stores it in a database.

[1026] 3. Synthesis request acceptance and emotion recognition

[1027] The user (data user) accesses the platform and creates a request for speech synthesis. The device provides a form where the user can select the provider of the voice data they want to use and input the content they want to generate (e.g., text message or lyrics). The device then sends the input text to emotion recognition software (e.g., Google Cloud Natural Language, IBM Watson), which analyzes emotions from the text.

[1028] Examples:

[1029] The user inputs the content they want to generate.

[1030] The device sends the input text to emotion recognition software, which analyzes the emotion.

[1031] 4. Speech synthesis and emotion-based adjustment

[1032] The emotion recognition software analyzes the emotional information and sends it to the server, which retrieves the corresponding voice data from the database and generates speech using a speech synthesis AI model (e.g., Tacotron, WaveNet). The generated speech is adjusted according to the emotion.

[1033] Examples:

[1034] The server acquires the emotion information and acquires the voice data from the database.

[1035] The server uses a speech synthesis AI model to generate speech that corresponds to the emotion.

[1036] 5. Provision of audio data

[1037] The server stores the generated voice data in a database and provides it to the data user's device. The data user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[1038] Examples:

[1039] The server stores the generated audio data in a database.

[1040] The device downloads the audio data.

[1041] Example prompt sentence:

[1042] "Using the voice of voice data provider A, synthesize the following text with a friendly emotion: 'Try this product.'"

[1043] 6. Monitoring and Alerts

[1044] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[1045] Examples:

[1046] The server monitors the access log and detects any abnormalities.

[1047] The server sends an alert notification to the user.

[1048] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1049] Step 1:

[1050] The user installs the dedicated app on their device and launches it. When the user presses the record button, the device starts recording the audio. After recording is complete, the device accepts metadata entered by the user (such as name, age, gender, and audio description).

[1051] Input: User starts recording and enters metadata

[1052] Output: Recorded audio data and metadata

[1053] Specific behavior:

[1054] The user presses the record button.

[1055] The device will record and temporarily store the audio.

[1056] The user enters the required metadata.

[1057] Step 2:

[1058] The device sends the recorded voice data and input metadata to a server, which then analyzes the received voice data using voice analysis software, extracts features (pitch, tone, timbre, etc.), and stores them in a database.

[1059] Input: Audio data and metadata sent from the device

[1060] Output: Analyzed feature data

[1061] Specific behavior:

[1062] The device encrypts the audio data and metadata and sends them to the server.

[1063] The server reads the voice data and extracts features using analysis software.

[1064] The feature data is stored in a database.

[1065] Step 3:

[1066] The user logs in to the platform and sets the terms of use for the audio data. They enter the terms of use (whether commercial use is permitted, the usage fee, etc.) in a form displayed on the device, and the device sends this information to the server. The server updates the rights management information in the database based on the received terms and conditions and generates a usage policy.

[1067] Input: Terms of Use Information

[1068] Output: Updated rights management information and usage policy

[1069] Specific behavior:

[1070] The user logs into the platform and enters the terms of use.

[1071] The terminal transmits the usage conditions to the server.

[1072] The server updates the database based on the condition information and generates a usage policy.

[1073] Step 4:

[1074] The user (data user) accesses the platform and creates a request for voice synthesis. The device provides a form where the user can select the provider of the voice data they want to use and input the content they want to generate (text message or lyrics). The device sends the input text to emotion recognition software, which analyzes emotions from the text.

[1075] Input: The content (text) you want to generate

[1076] Output: Parsed emotion information

[1077] Specific behavior:

[1078] The user selects a provider and enters text content.

[1079] The device sends the text to emotion recognition software.

[1080] Emotion recognition software analyzes emotions from text and returns emotional information.

[1081] Step 5:

[1082] The emotion recognition software analyzes the emotional information and sends it to a server, which retrieves the corresponding voice data from a database and generates a voice using a voice synthesis AI model. The generated voice is adjusted according to the emotion.

[1083] Input: Emotional information, voice data

[1084] Output: Generated audio data

[1085] Specific behavior:

[1086] The server retrieves the audio data from the database.

[1087] The server inputs emotional information and voice data into the AI ​​model.

[1088] The AI ​​model generates voice based on the emotion.

[1089] Step 6:

[1090] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[1091] Input: Generated audio data

[1092] Output: Provided audio data, profits distributed

[1093] Specific behavior:

[1094] The server stores the generated audio data in a database.

[1095] The server sends a download link for the audio data to the device.

[1096] The device downloads the audio data.

[1097] The server calculates profits based on usage and distributes them to data providers.

[1098] Step 7:

[1099] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[1100] Input: Voice data usage

[1101] Output: Alert notification

[1102] Specific behavior:

[1103] The server monitors the access log and detects any abnormalities.

[1104] The server generates an alert and notifies the user.

[1105] (Application example 2)

[1106] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1107] While conventional speech synthesis technologies can generate voice data using a specific person's voice, they cannot automatically adjust the voice according to emotion, making it difficult to communicate naturally with users. Furthermore, there is a lack of effective systems for managing voice data usage rights and detecting unauthorized use. There is a need for a system that can recognize emotions and adjust the voice according to the user's text message in real time.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing usage rights of the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a providing means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, an emotion recognition means for analyzing emotions from an input text message, and a voice synthesis adjustment means for adjusting the voice based on the analyzed emotions. This makes it possible to use the voice data of a specific person and adjust the voice based on emotions corresponding to the text message, thereby realizing more human-like communication.

[1109] The "registration means" is a function for registering voice data of a specific person in the system.

[1110] The "rights management means" is a function for appropriately managing the rights to use audio data and protecting the rights of data providers.

[1111] The "request receiving means" is a function for receiving a request to synthesize voice data.

[1112] The "voice synthesis means" is a function for generating voice data based on a synthesis request.

[1113] The "provision means" is a function for providing synthesized voice data to the user.

[1114] The "monitoring means" is a function for continuously monitoring the usage status of voice data.

[1115] The "detection means" is a function for detecting unauthorized use of audio data.

[1116] The "notification means" is a function for sending an alert to the user when unauthorized use is detected.

[1117] The "emotion recognition means" is a function for analyzing emotions from a text message entered by a user.

[1118] The "voice synthesis adjustment means" is a function for adjusting the voice data to be generated based on the analyzed emotion.

[1119] A specific system configuration and its operation will be described below for an embodiment of the present invention.

[1120] 1. Registering audio data

[1121] The user installs a dedicated application on their smartphone. They launch the application and register voice data through an interface that allows them to record their own voice. The user presses the record button, records their own voice, and enters the necessary metadata (such as name, age, gender, and voice description). The device then sends the recorded voice data and metadata to the server. The server analyzes the voice data, extracts features, and stores them in a database.

[1122] 2. Rights Management

[1123] Users log in to the platform and set the terms of use for the audio data (for example, whether commercial use is permitted, and the usage fee setting). The setting information is sent to the server, which then updates the rights management information and generates a usage policy based on that information.

[1124] 3. Synthesis request acceptance and emotion recognition

[1125] A user (data user) accesses the platform and creates a request for voice synthesis. The user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device then activates the emotion recognition means and analyzes emotions from the input text.

[1126] 4. Speech synthesis and emotion-based adjustment

[1127] The emotion recognition means analyzes the text message and determines the emotion (e.g., joy, sadness, anger, etc.). This emotion information is sent to the server, which retrieves the corresponding voice data from a database. A speech synthesis AI model is used to generate a voice corresponding to the emotion. For example, for the input text "I'm so happy today!", the system detects the emotion "joy" and generates a voice with intonation and nuance that matches the emotion of joy.

[1128] 5. Provision of audio data

[1129] The server stores the generated voice data in a database and provides it to the user's device. The user downloads the provided voice data and uses it for the specified purpose.

[1130] 6. Monitoring and Alerts

[1131] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or use that violates the set policy, it generates an alert and notifies the user's device.

[1132] Hardware and software used

[1133] Hardware: Smartphones, servers

[1134] Software: Google's Natural Language API, Amazon Polly

[1135] Examples and prompts

[1136] Example user input: "My item arrived broken. What should I do?"

[1137] Prompt statement:

[1138] Please send the following text message to the Emotion Recognition API to analyze the emotion: My item arrived damaged. What should I do?

[1139] => Result: Anger

[1140] Please use the speech synthesis API to generate speech based on the following text message with the emotion "Anger." : Dear customer, we apologize for the inconvenience and will respond promptly.

[1141] => Audio data generation

[1142] Send the generated voice data to the customer

[1143] In this way, by using a specific person's voice data and adjusting the voice according to emotions based on text messages, more human-like communication becomes possible. In addition, rights management and fraudulent use monitoring functions ensure the rights of data providers are protected.

[1144] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1145] Step 1:

[1146] The user installs and launches a dedicated application on their smartphone. The user records voice data and enters metadata (such as name, age, gender, and a description of the voice). The device then sends the recorded voice data and the entered metadata to a server. The server analyzes the voice data, extracts features (such as pitch, tone, and sound quality), and stores them in a database.

[1147] Step 2:

[1148] The user logs in to the platform and sets the terms of use for the audio data. The terms of use set by the user (e.g., whether commercial use is permitted, the usage fee, etc.) are sent to the server via the terminal. The server updates the rights management information based on the received terms and conditions and stores it in a database.

[1149] Step 3:

[1150] The data user accesses the platform and creates a request for voice synthesis. The data user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device sends the input text message to the server, which calls the emotion recognition API to analyze the emotion. The server obtains the analyzed emotion and saves the results.

[1151] Step 4:

[1152] The server generates voice data using a speech synthesis AI model based on the text message and emotion recognition results. The generation AI model adjusts the intonation and nuance of the voice based on emotions (e.g., joy, sadness, anger) to create voice data. The generated voice data is stored on the server.

[1153] Step 5:

[1154] The server provides the generated voice data to the data user's device. The data user's device receives the voice data, and the user downloads and uses it. The server calculates profits based on the amount of voice data used and distributes them to the data provider.

[1155] Step 6:

[1156] The server continuously monitors the usage of voice data. To detect signs of fraudulent or unauthorized use, the monitoring system monitors for abnormal access or usage that violates established policies. If an abnormality is detected, the server generates an alert and notifies the data provider's device.

[1157] As described above, this system uses the voice data of a specific person to generate and provide voice that corresponds to the emotion of the text message, allowing users to communicate more humanly. It also manages the rights of the voice data and monitors unauthorized use, protecting the rights of the data provider.

[1158] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1159] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1160] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1161] [Fourth embodiment]

[1162] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1163] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1164] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1165] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1166] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1167] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1168] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1169] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1170] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1171] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1172] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1173] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1174] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1175] The present invention is a system for providing a speech synthesis service based on the speech data of a specific person, and includes elements of a user, a terminal, and a server. Below, we will explain how the elements of this system work together to realize specific functions.

[1176] 1. Registering audio data

[1177] The user (voice data provider) installs a dedicated app on their device to record their voice. The user launches the app and records the voice to be provided. They also enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality).

[1178] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data and stores it in a database. It extracts features of the audio data (e.g., pitch, tone, and sound quality), generates identification information, and records it in the database.

[1179] 2. Rights Management

[1180] Users log in to the platform and set the terms of use for their own voice data (whether commercial use is permitted, usage fees, etc.). The device displays a form for entering this setting information and sends the user's input to the server.

[1181] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[1182] 3. Accepting synthesis requests

[1183] A user (data user) makes a request through the platform to generate audio based on a specific person's voice. The device provides a form for the user to select the provider of the audio data they want to use and to input the content they want to generate (such as a text message or lyrics). The user enters the request details, and the device sends this information to the server.

[1184] 4. Speech Synthesis

[1185] The server retrieves the corresponding voice data from the database based on the received synthesis request. Using a voice synthesis AI model, it inputs the text provided by the user and generates a realistic voice in the voice of a specific person. The synthesized voice data is stored in the database on the server.

[1186] 5. Provision of audio data

[1187] The server provides the generated voice data to the data user's terminal. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[1188] 6. Monitoring and Alerts

[1189] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[1190] Specific examples

[1191] For example, voice data provider A records his or her voice and registers it on the platform. A sets conditions allowing only commercial use. Meanwhile, advertising production company B wants to create a narration for a product introduction using provider A's voice and sends a request. The synthetic AI generates a narration in provider A's voice saying, "Please try this product," and provides it to B. The generated voice is for commercial use, and the compensation is distributed to provider A. During this process, the server monitors all usage and checks for any unauthorized use.

[1192] This system protects the rights of voice data providers and ensures appropriate profit distribution. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to users.

[1193] The processing flow will be explained below.

[1194] Step 1:

[1195] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[1196] Step 2:

[1197] The user presses the record button to record their own voice. After the recording is complete, the device temporarily stores the recorded voice data and displays a form for inputting necessary metadata (such as name, age, gender, and voice description).

[1198] Step 3:

[1199] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[1200] Step 4:

[1201] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[1202] Step 5:

[1203] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[1204] Step 6:

[1205] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[1206] Step 7:

[1207] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[1208] Step 8:

[1209] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[1210] Step 9:

[1211] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[1212] Step 10:

[1213] The server receives the request, retrieves the corresponding voice data from a database, and then uses a speech synthesis AI model to input the specified text and generate speech in the voice of a specific person.

[1214] Step 11:

[1215] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[1216] Step 12:

[1217] The server calculates profits based on the amount of voice data used, and the calculated profits are distributed to the data providers and recorded in a database.

[1218] Step 13:

[1219] The server continuously monitors the usage of voice data to detect signs of unauthorized use and generates an alert if an abnormality is detected.

[1220] Step 14:

[1221] The server sends the generated alert to the data provider's terminal, which displays the alert notification so that the user can check the status.

[1222] Through the above steps, this system enables the appropriate management and utilization of voice data, protecting the rights of data providers and providing new value.

[1223] Example 1

[1224] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1225] In speech synthesis systems based on voice data, there is a need to efficiently and safely manage the voice data of specific individuals, prevent unauthorized use, and realize accurate and realistic voice synthesis. In addition, a mechanism is needed to protect the rights of voice data providers and distribute profits appropriately.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1227] In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing rights to use the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a provision means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, a means for encrypting and transmitting metadata associated with the recorded voice data, and a prompt sentence generation means for inputting the voice data and text into a voice synthesis model. This enables safe management and use of voice data, protection of the rights of voice data providers, and realization of realistic voice synthesis.

[1228] The "registration means" is a mechanism for recording the voice data of a specific person and registering it in a database together with necessary metadata.

[1229] The "rights management means" is a mechanism for managing the rights to use audio data, and for setting and updating the conditions of use and permission details.

[1230] The "request receiving means" is a mechanism for receiving a request for voice synthesis based on voice data and transmitting the content of the request to the server.

[1231] The "voice synthesis means" is a mechanism for generating voice using voice data of a specific person based on a synthesis request.

[1232] The "means for providing" is a mechanism for providing the generated voice data to the terminal of the data user.

[1233] "Monitoring measures" are mechanisms for continuously monitoring the usage of voice data and detecting fraudulent or unauthorized use.

[1234] The "detection means" is a mechanism for determining whether or not there is unauthorized use of audio data based on information from the monitoring means.

[1235] The "notification means" is a mechanism for notifying relevant parties of an alert when unauthorized use of audio data is detected.

[1236] "Means for encryption and transmission" refers to a mechanism for encrypting recorded audio data and associated metadata and transmitting them to a server using a secure communication protocol.

[1237] The "prompt sentence generation means" is a mechanism for generating a prompt sentence for inputting voice data and text into a voice synthesis model.

[1238] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and includes elements of a user, a terminal, and a server. This system has the necessary configuration to realize the functions of voice data registration, rights management, reception of synthesis requests, voice synthesis, voice data provision, monitoring, and alerts.

[1239] Users install a dedicated app on their device to record their own voice. Using this app, users record their voice and enter necessary profile information and metadata related to the voice data (such as name, age, gender, and voice quality). The device encrypts the recorded voice data and the entered metadata and sends them to the server using a security protocol such as TLS (Transport Layer Security). The server analyzes the received voice data using acoustic analysis software (e.g., Praat or Audacity), extracts voice features, and stores them in a database.

[1240] Next, the user logs in to the platform and sets the conditions for using the audio data (whether commercial use is permitted, usage fees, etc.). The terminal displays a form for entering this setting information and sends the user's input to the server. The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions.

[1241] When generating audio using audio data, the user (data user) selects an audio data provider through the platform and inputs the content they want to generate (such as a text message or lyrics). For example, they input the text "Please try this product." The device then sends this information to the server.

[1242] The server retrieves the corresponding voice data from the database and inputs the text provided by the user using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). Specifically, the prompt "Try this product" is input into the speech synthesis model. The server stores the generated voice data in the database and then provides it to the user's device. The user can download the provided voice data and use it for the specified purpose.

[1243] Furthermore, the server continuously monitors the usage of voice data to detect signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or behavior that violates usage policies, it generates an alert and notifies the data provider's device, allowing the user to take appropriate action.

[1244] For example, if voice data provider A records his or her voice and registers on the platform under the condition that only commercial use is permitted, and an advertising production company wishes to create a narration in A's voice saying, "Please try this product." The server that receives this request through the platform uses A's voice to generate a narration using a voice synthesis model and provides it to the advertising production company. The generated voice is used commercially, and the compensation is distributed to A.

[1245] This system protects the rights of voice data providers and ensures appropriate profit sharing. It also makes it possible to use voice synthesis technology to generate realistic voice content and provide it to each user.

[1246] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1247] Step 1: Record audio and enter metadata

[1248] The user installs a dedicated recording app on their device and launches it. The user presses the record button to record their own voice. Once recording is complete, the app displays a form in which the user can enter profile information (name, age, gender, voice quality, etc.). The data entered by the user here (audio data and metadata) is the input. The recorded audio data and the entered metadata are generated as the output.

[1249] Step 2: Encrypt and transmit audio data and metadata

[1250] The device encrypts the recorded audio data and input metadata using a security protocol such as TLS (Transport Layer Security). The encrypted audio data and metadata are input, and the output is sent to a server via a secure communication protocol.

[1251] Step 3: Analyze and save the audio data

[1252] The server decrypts the received encrypted audio data and metadata and analyzes it using acoustic analysis software (e.g., Praat or Audacity). This analysis extracts audio features (pitch, tone, timbre, etc.). The input is the decrypted audio data, and the output is the analyzed audio features and a process to store them in a database.

[1253] Step 4: Set the terms of use

[1254] The user logs in to a web or mobile platform and sets the terms of use for the audio data (whether commercial use is permitted, the usage fee, etc.). The terminal displays a form for inputting this setting information and accepts the user's input. The set terms of use are the input, and the terminal sends this information to the server. As an output, the rights management information in the database is updated.

[1255] Step 5: Accepting a synthesis request

[1256] The user (data user) selects a voice data provider and inputs the text of the content they want to generate. The input is the input text (e.g., a prompt sentence such as "Try this product"). The terminal sends this information to the server. The server receives a synthesis request as output.

[1257] Step 6: Acquiring audio data and synthesizing speech

[1258] The server retrieves the corresponding voice data from the database and inputs the text using a speech synthesis AI model (e.g., Tacotron 2 or WaveNet). The inputs are the retrieved voice data and the prompt. The speech synthesis AI model uses these to generate realistic speech. The output is the synthesized voice data, which is stored in the database.

[1259] Step 7: Provide synthetic voice data

[1260] The server provides the generated voice data to the user's (data user's) terminal. The provided voice data is the output. The user downloads this voice data and uses it for the specified purpose.

[1261] Step 8: Monitor usage and generate alerts

[1262] The server monitors the usage of voice data in real time. The input is the database access log. If the monitoring detects any fraudulent or unauthorized use, an alert is generated. The generated alert (notification message) is the output.

[1263] Step 9: Alert Notification

[1264] The server notifies the generated alert to the user's (data provider's) device. The alert notification (for example, app notification or email) is the output, allowing the user to take action against fraudulent use.

[1265] Through this series of steps, safe management and use of voice data, protection of the rights of voice data providers, and realistic voice synthesis are realized.

[1266] (Application example 1)

[1267] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1268] In conventional virtual stores, when users ask questions about products or services in real time, only text-based responses are provided, which poses a problem of lacking realism and reliability. Furthermore, even when using speech synthesis technology, there is a lack of a mechanism for reproducing a specific person's voice in real time, which limits the interactive experience. For this reason, there is a need for a system that can respond in real time using a specific person's voice.

[1269] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1270] In this invention, the server includes: a registration means for registering voice data of a specific person; a rights management means for managing usage rights of the voice data; a request receiving means for receiving a synthesis request; a voice synthesis means for synthesizing voice data based on the synthesis request; a provision means for providing the synthesized voice data; a monitoring means for monitoring the usage status of the voice data; a detection means for detecting unauthorized use of the voice data; a notification means for issuing an alert when unauthorized use is detected; a generation AI model utilization means for using a generation AI model when a user uses the voice synthesis service and synthesizing voice data in real time based on a prompt sentence; and a voice data provision means for providing the synthesized voice data to a terminal and receiving the voice data as a voice file, thereby enabling a user to receive explanations about products and services in real time in a virtual store from the voice of a specific person.

[1271] "Voice data of a specific person" is data that is a digital recording of a voice provided by a specific individual.

[1272] The "registration means" is a function that allows voice data providers to register their own voice data in the system.

[1273] The "rights management means" is a function that manages the conditions of use and rights information for audio data, and protects the rights of data providers.

[1274] The "request receiving means" is a function that allows a user of voice data to request voice synthesis using the voice of a specific person.

[1275] The "voice synthesis means" is a function for converting designated text into voice by using voice data of a specific person based on a request.

[1276] The "provision means" is a function for providing the generated voice data to the user.

[1277] "Monitoring means" is a function that monitors the usage of voice data and checks for any fraudulent or unauthorized use.

[1278] "Detection means" is a function for detecting unauthorized or improper use.

[1279] The "notification means" is a function for notifying the data provider of an alert when unauthorized use is detected.

[1280] "Means for utilizing a generative AI model" refers to a function for synthesizing voice data in real time based on a prompt sentence using a generative AI model.

[1281] The "voice data providing means" is a function that provides synthesized voice data to the terminal and allows the user to receive the voice data through a voice file.

[1282] This invention is a system for providing a voice synthesis service based on the voice data of a specific person, and is composed of a user, a terminal, and a server. This system includes functions for registering the voice data of a specific person, managing rights, accepting synthesis requests, voice synthesis, providing voice data, monitoring, detecting and notifying unauthorized use, utilizing a generative AI model, and providing data.

[1283] 1. Registering audio data

[1284] Users install an application on their device to record their own voice. They use this application to record voice data and enter metadata such as profile information and sound quality. The device then sends the recorded voice data and metadata to a server. The server analyzes the voice data and stores it in a database.

[1285] 2. Rights Management

[1286] The user logs in to the server and sets the conditions for using their own audio data. The terminal sends this information to the server. The server updates the rights management information in the database based on the condition information and generates a usage policy.

[1287] 3. Accepting synthesis requests

[1288] A user of voice data makes a request through a terminal to generate voice data based on the voice of a specific person. The terminal then selects the voice data provider they wish to use and provides a form for inputting the text message or question they wish to generate. This information is then sent to the server.

[1289] 4. Speech Synthesis

[1290] The server retrieves the corresponding voice data from the database based on the received synthesis request, and uses the generative AI model to synthesize a realistic voice with the voice of a specific person based on the prompt sentence. The generated voice data is stored in the database on the server.

[1291] 5. Provision of audio data

[1292] The server provides the generated voice data to the terminal, which the user can download and use for the specified purpose.

[1293] 6. Monitoring and Alerts

[1294] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. If unauthorized use is detected, the server generates an alert and notifies the data provider's device.

[1295] Specific examples

[1296] For example, if a user asks, "Please tell me about the ingredients in this new cosmetic product," the server uses a generative AI model to generate a response text. Based on this text, the server uses a speech synthesis API such as AWS Polly to generate voice data and respond in the voice of a specific person. This voice data is then provided to the user's device.

[1297] Example prompt sentence:

[1298] "Please tell me about the ingredients in this new cosmetic product."

[1299] In this way, by using this system, users can enjoy a realistic voice interaction experience and receive explanations about products and services in the virtual store.

[1300] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1301] Step 1:

[1302] Users use a dedicated application installed on their device to record their own voice and input metadata such as profile information and sound quality. Input: Voice data, profile information. Output: Recorded voice data, metadata.

[1303] Step 2:

[1304] The device sends the recorded audio data and input metadata to the server. Input: Recorded audio data, metadata. Output: Audio data and metadata sent to the server.

[1305] Step 3:

[1306] The server analyzes the received voice data and stores it in a voice database. At the same time, it extracts features from the voice data and records them in the database. Input: Voice data, metadata. Output: Stored voice data, features.

[1307] Step 4:

[1308] The user logs in to the server and sets the conditions for using their own audio data on the rights management screen. Input: Conditions of use (whether commercial use is permitted, usage fees, etc.). Output: The set conditions of use.

[1309] Step 5:

[1310] The terminal transmits the set usage conditions to the server, which records them in the rights management database. Input: Usage conditions. Output: Usage conditions recorded in the rights management database.

[1311] Step 6:

[1312] The user of the voice data makes a request through their terminal to generate voice based on the voice of a specific person. In the form, they enter the voice data provider they want to use and the text message they want to generate. Input: Voice data provider, text message. Output: Request details.

[1313] Step 7:

[1314] The terminal sends the request content to the server, and the server retrieves the corresponding voice data from the database. Input: Request content. Output: Retrieved voice data.

[1315] Step 8:

[1316] The server uses a generative AI model to convert text to speech in a specific person's voice based on the prompt and the acquired voice data. Input: Prompt, voice data. Output: Generated voice data.

[1317] Step 9:

[1318] The server stores the generated voice data in a database and provides it to the user's device. Input: Generated voice data. Output: Provided voice data.

[1319] Step 10:

[1320] The server continuously monitors the usage of voice data and detects abnormal access or usage that violates policy. Input: Usage data. Output: Monitoring reports, alerts.

[1321] Step 11:

[1322] If unauthorized use is detected, the server generates an alert and notifies the data provider's terminal. Input: Detection result. Output: Alert notification.

[1323] Step 12:

[1324] The user receives the voice data generated in real time through the terminal and uses it as an audio file. For example, if the question is "Please tell me about the ingredients of this new cosmetic product," the user will receive the answer in real time in a specific voice. Input: Provided voice data. Output: Playback of the audio file.

[1325] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1326] The present invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person, and its purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we will explain how the elements of this system work together to realize specific functions.

[1327] 1. Registering audio data

[1328] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input. The user presses the record button to record their own voice and enters the necessary metadata (name, age, gender, voice description, etc.).

[1329] The device sends the recorded audio data and input metadata to the server, which analyzes the received audio data, extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[1330] 2. Rights Management

[1331] The user logs in to the platform and sets the terms of use for their own voice data. The terminal displays a form that prompts the user to enter the terms of use (for example, whether commercial use is permitted, the usage fee, etc.). When the user enters the terms of use and presses the send button, the terminal sends the entered terms of use to the server.

[1332] The server updates the rights management information in the database based on the received condition information and generates a usage policy based on the usage conditions. This usage policy is used to verify whether the use of the audio data complies with the set conditions.

[1333] 3. Synthesis request acceptance and emotion recognition

[1334] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal provides a form in which the user selects a provider of the voice data they wish to use and inputs the content they wish to generate (e.g., text message or lyrics). The terminal then activates an emotion recognition means to recognize emotions from the user's input data and analyzes the emotions from the input text.

[1335] 4. Speech synthesis and emotion-based adjustment

[1336] The emotion recognition means analyzes emotions (e.g., joy, sadness, anger, etc.) from the input text and sends the emotion to the server, which retrieves the corresponding voice data from the database and generates voice using a voice synthesis AI model based on the user's emotion.

[1337] For example, if the input text is "I'm so happy today!", the emotion recognition means will determine this as "joy" and provide that information to the AI ​​model, which will then generate speech with intonation and nuances that match the emotion of joy.

[1338] 5. Provision of audio data

[1339] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[1340] 6. Monitoring and Alerts

[1341] The server continuously monitors the usage of voice data, watching for signs of fraudulent or unauthorized use. If the monitoring system detects abnormal access or usage that violates established policies, it generates an alert. The alert is notified to the data provider's device, and the user can check the situation through an interface to take appropriate action.

[1342] Specific examples

[1343] For example, voice data provider A records his or her voice and registers it on the platform. A sets a condition that only commercial use is permitted. Meanwhile, advertising production company B wants to create a narration for a product introduction in provider A's voice and sends a request. If the text provided by user B is "Please try this product," the emotion recognition means analyzes the text and recognizes a friendly emotion. Based on this emotional information, the server generates a narration in provider A's voice with a friendly tone. The generated voice is provided to advertising production company B, and the compensation is distributed to provider A.

[1344] This system protects the rights of voice data providers and enables high-quality voice synthesis that reflects emotions, thereby providing new value.

[1345] The processing flow will be explained below.

[1346] Step 1:

[1347] The user (data provider) installs the dedicated app on their device and launches it. The device displays an interface for recording the user's voice and accepts voice input.

[1348] Step 2:

[1349] The user presses the record button to record their own voice. Once the recording is complete, the device temporarily stores the recorded voice data and displays a form for entering necessary metadata (such as name, age, gender, and voice description).

[1350] Step 3:

[1351] The user enters the required metadata into the form and presses the submit button. The terminal then sends the recorded audio data and the entered metadata to the server.

[1352] Step 4:

[1353] The server receives the audio data and metadata sent from the device, analyzes the audio data, and extracts features (e.g., pitch, tone, and timbre). This data is then stored in a database.

[1354] Step 5:

[1355] The user logs in to the platform and sets the conditions for using their own voice data. The terminal displays a form for the user to enter the conditions for use (e.g., whether commercial use is permitted, usage fees, etc.).

[1356] Step 6:

[1357] The user enters the usage conditions into the form and presses the send button. The terminal then sends the entered usage conditions to the server.

[1358] Step 7:

[1359] The server receives the usage conditions information, updates the rights management information in the database, and generates and applies usage policies based on the usage conditions.

[1360] Step 8:

[1361] A user (data user) accesses the platform and creates a request for voice synthesis. The terminal displays a form in which the user can select the provider of the voice data they want to use and input the content they want to generate (for example, a text message or lyrics).

[1362] Step 9:

[1363] The user inputs the request details and presses the send button. The terminal then sends the selected provider's ID and the request text to the server.

[1364] Step 10:

[1365] The server receives the request and retrieves the corresponding voice data from the database. Then, it uses an emotion recognition unit to analyze emotions based on the request text. The emotion recognition unit analyzes keywords and context within the text to identify specific emotions (e.g., joy, sadness, anger, etc.).

[1366] Step 11:

[1367] The server uses a speech synthesis AI model based on the analyzed emotional information to generate an appropriate voice from the voice data of a specific person. For example, if the text says "I'm so happy today!", the emotion recognition means will identify "joy" and the speech synthesis AI will generate a voice that expresses "joy."

[1368] Step 12:

[1369] The server stores the generated voice data in a database and sends it to the data user's terminal. The terminal receives the synthesized voice data and displays and saves it in the format specified by the user (data user).

[1370] Step 13:

[1371] The server calculates profits based on the amount of voice data used, and records the calculated profits to be distributed to users (data providers).

[1372] Step 14:

[1373] The server continuously monitors voice data usage to detect signs of abuse and generates an alert if unauthorized use is detected.

[1374] Step 15:

[1375] The server sends the generated alert to the user's (data provider's) terminal, which displays the alert notification so that the user can check the status.

[1376] By combining emotion recognition and speech synthesis technologies, this system can generate and provide high-quality audio content that matches the user's emotions, thereby protecting the rights of audio data providers.

[1377] Example 2

[1378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] Conventional speech synthesis systems have difficulty synthesizing a specific person's voice naturally, and speech synthesis based on emotion recognition in particular has not been fully realized. Furthermore, managing the rights to use voice data and monitoring and detecting unauthorized use are difficult, and the rights of data providers are not adequately protected. Therefore, there is a need for a system that not only provides high-quality speech synthesis but also reliably protects the rights of data providers.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1381] In this invention, the server includes a registration means for registering voice data of a specific person through a dedicated application installed on the user's device, an analysis means for analyzing the voice data and extracting features, a rights management means for allowing the user to set usage conditions for the voice data, a voice synthesis means for analyzing emotions from input text using an emotion recognition means and synthesizing voice based on the emotions, a provision means for storing the synthesized voice data in a database and providing it, a monitoring means for continuously monitoring the usage status of the voice data, and a notification means for detecting unauthorized use of the voice data and notifying the user. This not only enables high-quality voice synthesis and natural-sounding voice generation based on the user's emotions, but also protects the rights of data providers and makes it easy to monitor and detect unauthorized use.

[1382] A "dedicated app" is software with specific functions that is installed on a user's device and used.

[1383] "Voice data" refers to acoustic information that is a recording of a specific person's voice and stored in digital format.

[1384] "Features" are characteristic values ​​such as pitch, tone, and sound quality extracted from audio data.

[1385] "Rights management" refers to the procedures and systems for setting and managing the terms of use and rights information for audio data.

[1386] An "emotion recognition means" is a technology or tool that analyzes and identifies a user's emotions from input text or voice.

[1387] "Speech synthesis means" refers to technology and devices that generate new voices based on specific voice data and emotional information.

[1388] "Providing means" refers to the process and system for providing synthesized voice data to the user.

[1389] "Monitoring Measures" are technologies and tools that continuously monitor the usage of voice data.

[1390] The "notification means" refers to a system and method for notifying the user when unauthorized use or abnormality of audio data is detected.

[1391] The "profit distribution means" is a method and system for calculating profits according to the amount of use of voice data and distributing them to data providers.

[1392] A "voice synthesis AI model" is a model that uses artificial intelligence to generate high-quality voice based on voice data and emotional information.

[1393] This invention is a system that provides a service that combines speech synthesis and emotion recognition functions based on the voice data of a specific person. Its particular purpose is to recognize the user's emotions and synthesize speech in accordance with those emotions. Below, we explain how the elements of this system work together to realize specific functions.

[1394] 1. Registering audio data

[1395] The user installs a dedicated app on their device and launches it. The app has an interface for recording audio, and the user presses the record button to begin recording. When recording is complete, the user enters metadata such as their name, age, gender, and a description of the audio. The device then sends the recorded audio data and the entered metadata to a server. The server then analyzes the received audio data using audio analysis software (e.g., Praat, LibROSA), extracts features (e.g., pitch, tone, timbre, etc.), and stores them in a database.

[1396] Examples:

[1397] When the user presses the record button, the device starts recording.

[1398] After recording, the user enters the metadata and sends it to the server.

[1399] The server analyzes the voice data and stores the features in a database.

[1400] 2. Rights Management

[1401] The user logs in to the platform and sets the terms of use for the audio data. The device displays a form for entering the terms of use (e.g., whether commercial use is permitted, and the usage fee). The user enters the terms of use, and the device sends the information to the server. The server updates the rights management information in the database based on the received terms and generates a usage policy based on the terms of use.

[1402] Examples:

[1403] The user logs into the platform and enters the terms of use.

[1404] The terminal transmits the entered usage conditions to the server.

[1405] The server generates a usage policy and stores it in a database.

[1406] 3. Synthesis request acceptance and emotion recognition

[1407] The user (data user) accesses the platform and creates a request for speech synthesis. The device provides a form where the user can select the provider of the voice data they want to use and input the content they want to generate (e.g., text message or lyrics). The device then sends the input text to emotion recognition software (e.g., Google Cloud Natural Language, IBM Watson), which analyzes emotions from the text.

[1408] Examples:

[1409] The user inputs the content they want to generate.

[1410] The device sends the input text to emotion recognition software, which analyzes the emotion.

[1411] 4. Speech synthesis and emotion-based adjustment

[1412] The emotion recognition software analyzes the emotional information and sends it to the server, which retrieves the corresponding voice data from the database and generates speech using a speech synthesis AI model (e.g., Tacotron, WaveNet). The generated speech is adjusted according to the emotion.

[1413] Examples:

[1414] The server acquires the emotion information and acquires the voice data from the database.

[1415] The server uses a speech synthesis AI model to generate speech that corresponds to the emotion.

[1416] 5. Provision of audio data

[1417] The server stores the generated voice data in a database and provides it to the data user's device. The data user can download the provided voice data and use it for the specified purpose. The server also calculates profits according to the amount of voice data used and distributes them to the data provider.

[1418] Examples:

[1419] The server stores the generated audio data in a database.

[1420] The device downloads the audio data.

[1421] Example prompt sentence:

[1422] "Using the voice of voice data provider A, synthesize the following text with a friendly emotion: 'Try this product.'"

[1423] 6. Monitoring and Alerts

[1424] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[1425] Examples:

[1426] The server monitors the access log and detects any abnormalities.

[1427] The server sends an alert notification to the user.

[1428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1429] Step 1:

[1430] The user installs the dedicated app on their device and launches it. When the user presses the record button, the device starts recording the audio. After recording is complete, the device accepts metadata entered by the user (such as name, age, gender, and audio description).

[1431] Input: User starts recording and enters metadata

[1432] Output: Recorded audio data and metadata

[1433] Specific behavior:

[1434] The user presses the record button.

[1435] The device will record and temporarily store the audio.

[1436] The user enters the required metadata.

[1437] Step 2:

[1438] The device sends the recorded voice data and input metadata to a server, which then analyzes the received voice data using voice analysis software, extracts features (pitch, tone, timbre, etc.), and stores them in a database.

[1439] Input: Audio data and metadata sent from the device

[1440] Output: Analyzed feature data

[1441] Specific behavior:

[1442] The device encrypts the audio data and metadata and sends them to the server.

[1443] The server reads the voice data and extracts features using analysis software.

[1444] The feature data is stored in a database.

[1445] Step 3:

[1446] The user logs in to the platform and sets the terms of use for the audio data. They enter the terms of use (whether commercial use is permitted, the usage fee, etc.) in a form displayed on the device, and the device sends this information to the server. The server updates the rights management information in the database based on the received terms and conditions and generates a usage policy.

[1447] Input: Terms of Use Information

[1448] Output: Updated rights management information and usage policy

[1449] Specific behavior:

[1450] The user logs into the platform and enters the terms of use.

[1451] The terminal transmits the usage conditions to the server.

[1452] The server updates the database based on the condition information and generates a usage policy.

[1453] Step 4:

[1454] The user (data user) accesses the platform and creates a request for voice synthesis. The device provides a form where the user can select the provider of the voice data they want to use and input the content they want to generate (text message or lyrics). The device sends the input text to emotion recognition software, which analyzes emotions from the text.

[1455] Input: The content (text) you want to generate

[1456] Output: Parsed emotion information

[1457] Specific behavior:

[1458] The user selects a provider and enters text content.

[1459] The device sends the text to emotion recognition software.

[1460] Emotion recognition software analyzes emotions from text and returns emotional information.

[1461] Step 5:

[1462] The emotion recognition software analyzes the emotional information and sends it to a server, which retrieves the corresponding voice data from a database and generates a voice using a voice synthesis AI model. The generated voice is adjusted according to the emotion.

[1463] Input: Emotional information, voice data

[1464] Output: Generated audio data

[1465] Specific behavior:

[1466] The server retrieves the audio data from the database.

[1467] The server inputs emotional information and voice data into the AI ​​model.

[1468] The AI ​​model generates voice based on the emotion.

[1469] Step 6:

[1470] The server stores the generated voice data in a database and provides it to the data user's device. The user can download the provided voice data and use it for the specified purpose. The server also calculates profits based on the amount of voice data used and distributes them to the data provider.

[1471] Input: Generated audio data

[1472] Output: Provided audio data, profits distributed

[1473] Specific behavior:

[1474] The server stores the generated audio data in a database.

[1475] The server sends a download link for the audio data to the device.

[1476] The device downloads the audio data.

[1477] The server calculates profits based on usage and distributes them to data providers.

[1478] Step 7:

[1479] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or usage that violates the set policy, it generates an alert and notifies the user. The user receives the notification, checks the situation through the interface, and can take appropriate action.

[1480] Input: Voice data usage

[1481] Output: Alert notification

[1482] Specific behavior:

[1483] The server monitors the access log and detects any abnormalities.

[1484] The server generates an alert and notifies the user.

[1485] (Application example 2)

[1486] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1487] While conventional speech synthesis technologies can generate voice data using a specific person's voice, they cannot automatically adjust the voice according to emotion, making it difficult to communicate naturally with users. Furthermore, there is a lack of effective systems for managing voice data usage rights and detecting unauthorized use. There is a need for a system that can recognize emotions and adjust the voice according to the user's text message in real time.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a registration means for registering voice data of a specific person, a rights management means for managing usage rights of the voice data, a request receiving means for receiving a synthesis request, a voice synthesis means for synthesizing voice data based on the synthesis request, a providing means for providing the synthesized voice data, a monitoring means for monitoring the usage status of the voice data, a detection means for detecting unauthorized use of the voice data, a notification means for issuing an alert when unauthorized use is detected, an emotion recognition means for analyzing emotions from an input text message, and a voice synthesis adjustment means for adjusting the voice based on the analyzed emotions. This makes it possible to use the voice data of a specific person and adjust the voice based on emotions corresponding to the text message, thereby realizing more human-like communication.

[1489] The "registration means" is a function for registering voice data of a specific person in the system.

[1490] The "rights management means" is a function for appropriately managing the rights to use audio data and protecting the rights of data providers.

[1491] The "request receiving means" is a function for receiving a request to synthesize voice data.

[1492] The "voice synthesis means" is a function for generating voice data based on a synthesis request.

[1493] The "provision means" is a function for providing synthesized voice data to the user.

[1494] The "monitoring means" is a function for continuously monitoring the usage status of voice data.

[1495] The "detection means" is a function for detecting unauthorized use of audio data.

[1496] The "notification means" is a function for sending an alert to the user when unauthorized use is detected.

[1497] The "emotion recognition means" is a function for analyzing emotions from a text message entered by a user.

[1498] The "voice synthesis adjustment means" is a function for adjusting the voice data to be generated based on the analyzed emotion.

[1499] A specific system configuration and its operation will be described below for an embodiment of the present invention.

[1500] 1. Registering audio data

[1501] The user installs a dedicated application on their smartphone. They launch the application and register voice data through an interface that allows them to record their own voice. The user presses the record button, records their own voice, and enters the necessary metadata (such as name, age, gender, and voice description). The device then sends the recorded voice data and metadata to the server. The server analyzes the voice data, extracts features, and stores them in a database.

[1502] 2. Rights Management

[1503] Users log in to the platform and set the terms of use for the audio data (for example, whether commercial use is permitted, and the usage fee setting). The setting information is sent to the server, which then updates the rights management information and generates a usage policy based on that information.

[1504] 3. Synthesis request acceptance and emotion recognition

[1505] A user (data user) accesses the platform and creates a request for voice synthesis. The user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device then activates the emotion recognition means and analyzes emotions from the input text.

[1506] 4. Speech synthesis and emotion-based adjustment

[1507] The emotion recognition means analyzes the text message and determines the emotion (e.g., joy, sadness, anger, etc.). This emotion information is sent to the server, which retrieves the corresponding voice data from a database. A speech synthesis AI model is used to generate a voice corresponding to the emotion. For example, for the input text "I'm so happy today!", the system detects the emotion "joy" and generates a voice with intonation and nuance that matches the emotion of joy.

[1508] 5. Provision of audio data

[1509] The server stores the generated voice data in a database and provides it to the user's device. The user downloads the provided voice data and uses it for the specified purpose.

[1510] 6. Monitoring and Alerts

[1511] The server continuously monitors the usage of voice data, looking for signs of fraudulent or unauthorized use. If it detects abnormal access or use that violates the set policy, it generates an alert and notifies the user's device.

[1512] Hardware and software used

[1513] Hardware: Smartphones, servers

[1514] Software: Google's Natural Language API, Amazon Polly

[1515] Examples and prompts

[1516] Example user input: "My item arrived broken. What should I do?"

[1517] Prompt statement:

[1518] Please send the following text message to the Emotion Recognition API to analyze the emotion: My item arrived damaged. What should I do?

[1519] => Result: Anger

[1520] Please use the speech synthesis API to generate speech based on the following text message with the emotion "Anger." : Dear customer, we apologize for the inconvenience and will respond promptly.

[1521] => Audio data generation

[1522] Send the generated voice data to the customer

[1523] In this way, by using a specific person's voice data and adjusting the voice according to emotions based on text messages, more human-like communication becomes possible. In addition, rights management and fraudulent use monitoring functions ensure the rights of data providers are protected.

[1524] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1525] Step 1:

[1526] The user installs and launches a dedicated application on their smartphone. The user records voice data and enters metadata (such as name, age, gender, and a description of the voice). The device then sends the recorded voice data and the entered metadata to a server. The server analyzes the voice data, extracts features (such as pitch, tone, and sound quality), and stores them in a database.

[1527] Step 2:

[1528] The user logs in to the platform and sets the terms of use for the audio data. The terms of use set by the user (e.g., whether commercial use is permitted, the usage fee, etc.) are sent to the server via the terminal. The server updates the rights management information based on the received terms and conditions and stores it in a database.

[1529] Step 3:

[1530] The data user accesses the platform and creates a request for voice synthesis. The data user selects the provider of the voice data they wish to use and inputs the content they wish to generate (e.g., a text message). The device sends the input text message to the server, which calls the emotion recognition API to analyze the emotion. The server obtains the analyzed emotion and saves the results.

[1531] Step 4:

[1532] The server generates voice data using a speech synthesis AI model based on the text message and emotion recognition results. The generation AI model adjusts the intonation and nuance of the voice based on emotions (e.g., joy, sadness, anger) to create voice data. The generated voice data is stored on the server.

[1533] Step 5:

[1534] The server provides the generated voice data to the data user's device. The data user's device receives the voice data, and the user downloads and uses it. The server calculates profits based on the amount of voice data used and distributes them to the data provider.

[1535] Step 6:

[1536] The server continuously monitors the usage of voice data. To detect signs of fraudulent or unauthorized use, the monitoring system monitors for abnormal access or usage that violates established policies. If an abnormality is detected, the server generates an alert and notifies the data provider's device.

[1537] As described above, this system uses the voice data of a specific person to generate and provide voice that corresponds to the emotion of the text message, allowing users to communicate more humanly. It also manages the rights of the voice data and monitors unauthorized use, protecting the rights of the data provider.

[1538] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1540] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1541] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1542] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1543] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1544] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1545] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1546] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1547] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1548] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1549] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1550] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1551] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1552] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1553] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1554] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1555] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1556] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1557] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1558] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1559] The following is further disclosed regarding the above embodiment.

[1560] (Claim 1)

[1561] a registration means for registering voice data of a specific person;

[1562] a rights management means for managing rights to use the audio data;

[1563] a request receiving means for receiving a synthesis request;

[1564] a voice synthesis means for synthesizing voice data based on a synthesis request;

[1565] providing means for providing synthesized voice data;

[1566] a monitoring means for monitoring the usage of the voice data;

[1567] detection means for detecting unauthorized use of the audio data;

[1568] a notification means for providing an alert when unauthorized use is detected;

[1569] A system including:

[1570] (Claim 2)

[1571] 2. The system according to claim 1, further comprising profit sharing means for sharing profits for use of the voice data with the data provider.

[1572] (Claim 3)

[1573] The system of claim 1, which uses a voice synthesis AI model to realistically reproduce the voice of a specific person based on voice data.

[1574] "Example 1"

[1575] (Claim 1)

[1576] a registration means for registering voice data of a specific person;

[1577] a rights management means for managing rights to use the audio data;

[1578] a request receiving means for receiving a synthesis request;

[1579] a voice synthesis means for synthesizing voice data based on a synthesis request;

[1580] providing means for providing synthesized voice data;

[1581] a monitoring means for monitoring the usage of the voice data;

[1582] detection means for detecting unauthorized use of the audio data;

[1583] a notification means for providing an alert when unauthorized use is detected;

[1584] means for encrypting and transmitting the recorded audio data and associated metadata;

[1585] a prompt generation means for inputting speech data and text into a speech synthesis model;

[1586] A system including:

[1587] (Claim 2)

[1588] 2. The system according to claim 1, further comprising profit sharing means for sharing profits for use of the voice data with the data provider.

[1589] (Claim 3)

[1590] The system of claim 1, which uses a voice synthesis AI model to realistically reproduce the voice of a specific person based on voice data.

[1591] "Application Example 1"

[1592] (Claim 1)

[1593] a registration means for registering voice data of a specific person;

[1594] a rights management means for managing rights to use the audio data;

[1595] a request receiving means for receiving a synthesis request;

[1596] a voice synthesis means for synthesizing voice data based on a synthesis request;

[1597] providing means for providing synthesized voice data;

[1598] a monitoring means for monitoring the usage of the voice data;

[1599] detection means for detecting unauthorized use of the audio data;

[1600] a notification means for providing an alert when unauthorized use is detected;

[1601] a means for utilizing the generative AI model when a user uses the speech synthesis service, and for synthesizing speech data in real time based on a prompt sentence;

[1602] a voice data providing means for providing synthesized voice data to a terminal and for a user to receive the voice data through a voice file;

[1603] A system including:

[1604] (Claim 2)

[1605] 2. The system according to claim 1, further comprising profit sharing means for sharing profits for use of the voice data with the data provider.

[1606] (Claim 3)

[1607] The system of claim 1 uses a speech synthesis AI model to realistically reproduce the voice of a specific person based on voice data, generates a prompt sentence based on the user's question text, and synthesizes and provides the response as an audio file.

[1608] "Example 2: Combining Emotion Engines"

[1609] (Claim 1)

[1610] A registration means for registering voice data of a specific person through a dedicated application installed on a user's device;

[1611] analysis means for analyzing the voice data and extracting features;

[1612] a rights management means for allowing a user to set usage conditions for the audio data;

[1613] a speech synthesis means for analyzing emotions from input text using an emotion recognition means and synthesizing speech based on the emotion;

[1614] a providing means for storing the synthesized voice data in a database and providing the data;

[1615] a monitoring means for continuously monitoring the usage of the voice data;

[1616] a notification means for detecting unauthorized use of the audio data and notifying the user;

[1617] A system including:

[1618] (Claim 2)

[1619] 2. The system according to claim 1, further comprising profit distribution means for distributing profits to data providers in accordance with the amount of use of the voice data.

[1620] (Claim 3)

[1621] The system of claim 1 uses a voice synthesis AI model to realistically reproduce the voice of a specific person based on voice data and synthesize voice according to emotions.

[1622] "Application example 2 when combining emotion engines"

[1623] (Claim 1)

[1624] a registration means for registering voice data of a specific person;

[1625] a rights management means for managing rights to use the audio data;

[1626] a request receiving means for receiving a synthesis request;

[1627] a voice synthesis means for synthesizing voice data based on a synthesis request;

[1628] providing means for providing synthesized voice data;

[1629] a monitoring means for monitoring the usage of the voice data;

[1630] detection means for detecting unauthorized use of the audio data;

[1631] a notification means for providing an alert when unauthorized use is detected;

[1632] emotion recognition means for analyzing emotions from an input text message;

[1633] a speech synthesis adjusting means for adjusting the speech based on the analyzed emotion;

[1634] A system including:

[1635] (Claim 2)

[1636] 2. The system according to claim 1, further comprising profit sharing means for sharing profits for use of the voice data with the data provider.

[1637] (Claim 3)

[1638] The system of claim 1, which uses a voice synthesis AI model to realistically reproduce the voice of a specific person based on voice data. [Explanation of symbols]

[1639] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a registration means for registering voice data of a specific person; a rights management means for managing rights to use the audio data; a request receiving means for receiving a synthesis request; a voice synthesis means for synthesizing voice data based on a synthesis request; providing means for providing synthesized voice data; a monitoring means for monitoring the usage of the voice data; detection means for detecting unauthorized use of the audio data; a notification means for providing an alert when unauthorized use is detected; A system including:

2. 2. The system according to claim 1, further comprising profit sharing means for sharing profits for use of the voice data with the data provider.

3. 10. The system of claim 1, which uses a voice synthesis AI model to realistically reproduce the voice of a specific person based on voice data.

Images