system

Abstract

To provide a system which transmits user's emotion with improved accuracy.SOLUTION: A system comprises: means for detecting a feature of user's conversation using a generative AI model; deletion means for deleting the feature contained in the conversation as a noise, by inputting a specific prompt text into the generative AI model; means for generating processed sound not containing the feature removed as a noise, as noiseless clear sound; means for transmitting the processed sound to a listener listening to the conversation; and means for dynamically changing a phrase and a word to be deleted from the conversation based on an emotion engine recognizing user's emotion and the emotion.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Users have characteristic habits and phrases in their conversations, which can act as noise and hinder the transmission of information. In particular, when a service representative wants to anonymously ask employees for user opinions, the user's characteristic habits and phrases can undermine that anonymity. [Means for solving the problem]

[0005] The system of the present disclosure includes: means for detecting characteristics of a user's conversation using a generative AI model; deletion means for deleting the characteristics included in the conversation as noise by inputting a specific prompt sentence into the generative AI model; means for generating processed audio that does not include the characteristics deleted as noise as clear audio that does not include the noise; means for transmitting the processed audio to a listener who hears the conversation; an emotion engine that recognizes the user's emotions; and means for dynamically changing the selection of phrases and words to be deleted from the conversation based on the emotions. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One aspect of the present invention is a system that uses a generative AI to analyze a user's conversation in real time. This system detects features of the user's conversation, such as phrases like "umm" and "well then," and removes them as noise. The removal occurs the moment the generative AI detects a characteristic phrase, resulting in the generation of processed speech that does not include the feature.

[0029] The generated processed voice is sent to the other party through the system.

[0030] "Example 2"

[0031] As a specific example, a service representative may want to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the user's speech characteristics. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask for the user's opinion without losing their anonymity due to the user's distinctive mannerisms or phrases.

[0032] The processing flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[0035] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[0036] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[0037] Step 4: A processed voice without any features is generated and sent to the other party through the system.

[0038] "Example 2"

[0039] Step 1: When a service representative wants to anonymously ask a user for their opinion, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[0040] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[0041] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[0042] Step 4: A processed voice without the user's characteristics is generated and sent to the service representative. This allows the service representative to listen to the user's opinions anonymously and prevents the user's characteristic habits and phrases from compromising the anonymity.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] Conventional voice communication systems have a problem in that characteristic phrases such as "umm" and "well then" contained in user conversations remain as noise, making the voice difficult for the other party to hear. In addition, manually removing these noises requires a great deal of effort, making it impractical for real-time conversations. Furthermore, even when collecting user opinions anonymously, the presence of noise makes it difficult to accurately grasp their opinions.

[0046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0047] In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, and means for generating processed speech that does not include the removed characteristics. This enables noise to be removed from the user's conversation in real time to generate clear speech. The server also includes means for detecting characteristics of the user's conversation using a generative AI model, means for removing the characteristics included in the conversation as noise by inputting a specific prompt sentence into the generative AI model, means for generating processed speech that does not include the characteristics removed as noise as clear speech that does not include the noise, means for transmitting the processed speech to a listener who listens to the conversation, an emotion engine that recognizes the user's emotions, and means for dynamically changing the selection of phrases and words to be removed from the conversation based on the emotions. The removal means removes the characteristics included in the conversation as noise during live streaming over the Internet. The noise is a characteristic habit or phrase in the conversation that may compromise the user's anonymity.

[0048] "User" refers to an individual or entity that uses the system to engage in conversations.

[0049] "Conversational features" refer to specific phrases or words such as "um" or "well then" that are included in the user's speech.

[0050] "Noise" refers to unnecessary phrases and words that are detected as characteristics of conversation and are targeted for removal.

[0051] "Processed audio" refers to clear audio data after noise has been removed.

[0052] "Server" refers to a computer system for processing and analyzing voice data.

[0053] "Terminal" refers to a device used by a user that records and plays audio.

[0054] "Speech recognition software" refers to a program for converting voice data into text data.

[0055] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[0056] "Text data" refers to textual information converted by voice recognition software.

[0057] "Clean text data" refers to text data after noise has been removed.

[0058] "Voice data" refers to data that is a digital representation of a user's speech.

[0059] "Real-time" refers to processing occurring simultaneously with the user's conversation.

[0060] The present invention relates to a system for analyzing user conversations in real time and removing noise. Specific embodiments of this system will be described below.

[0061] System configuration

[0062] Hardware

[0063] Server: Uses a computer system equipped with high-performance CPUs and GPUs, which allows for large amounts of data processing and the execution of generative AI models.

[0064] Device: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[0065] software

[0066] Speech Recognition Software: Use a common speech recognition API (e.g., Google® Speech-to-Text API) to convert voice data into text data.

[0067] Generative AI models: Use generative AI models (e.g., OpenAI's GPT-3®) to analyze user conversations and detect characteristic phrases.

[0068] System Operation

[0069] Acquiring and sending audio data

[0070] When a user starts talking, the device uses a built-in microphone to record the user's voice, and the recorded voice data is sent to the server in real time.

[0071] Converting audio data to text

[0072] The server uses speech recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted directly into text data.

[0073] Text data analysis and noise removal

[0074] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. These phrases are removed as noise, and clean text data is generated.

[0075] Speech conversion of clean text data

[0076] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[0077] Sending and playing processed audio

[0078] The generated processed voice is sent from the server to the terminal, which then plays it back, allowing the user to hear clearer voice with the noise removed.

[0079] Examples of specific examples and prompts

[0080] For example, suppose a user has the following conversation:

[0081] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0082] The server receives this conversation and converts it into the following text data using speech recognition software:

[0083] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0084] Next, we use a generative AI model to remove noise as follows:

[0085] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[0086] Finally, the server converts the cleaned text data back into audio data, generates processed audio, and sends it to the terminal, which plays the processed audio.

[0087] An example of a prompt sentence might be:

[0088] "Detect distinctive phrases in user conversations and remove them as noise, such as 'um' or 'Well then.'"

[0089] By inputting this prompt into a generative AI model, a process is performed to remove noise from the user's conversation.

[0090] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0091] Step 1:

[0092] A user initiates a conversation.

[0093] The user begins speaking to the system, for example, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[0094] Step 2:

[0095] The device records the user's voice and sends it to the server.

[0096] The device records the user's voice using a built-in microphone. The recorded voice data is sent to the server in real time. The input is the user's voice data, and the output is the voice data sent to the server.

[0097] Step 3:

[0098] The server uses voice recognition software to convert the voice data into text data.

[0099] The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[0100] Step 4:

[0101] The server uses the generative AI model to analyze the text data and detect characteristic phrases.

[0102] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[0103] Step 5:

[0104] The server removes the detected phrases as noise and generates clean text data.

[0105] The server removes the detected characteristic phrases as noise. For example, from the text data "Um, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Um" and "So, let's go for a walk in the afternoon" to generate clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[0106] Step 6:

[0107] The server reconverts the clean text data into audio data.

[0108] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk this afternoon" is generated. The input is cleaned text data, and the output is speech data.

[0109] Step 7:

[0110] The server transmits the generated processed voice to the terminal.

[0111] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[0112] Step 8:

[0113] The device plays the processed audio.

[0114] The device then plays back the processed audio, allowing the user to hear clear audio with the noise removed. The input is the audio data sent to the device, and the output is the audio played back from the device's speaker.

[0115] (Application example 1)

[0116] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] Conventional audio distribution systems have the problem that noises such as "umm" and "well then" that occur during a streamer's conversation are transmitted to the listener, degrading the quality of the audio. Streamers are also required to consciously speak to avoid the noise, making it difficult to have a natural conversation.

[0118] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed audio that does not include the removed features. This makes it possible to remove noise from conversation in real time during live streaming or audio distribution. Furthermore, by including means for generating new text based on text from which noise has been removed using a generative AI model, the broadcaster can provide clear audio to listeners while continuing a natural conversation without having to consciously avoid noise.

[0119] "User speech characteristics" refers to specific phrases and word patterns that a user uses when speaking.

[0120] "Noise" refers to phrases or words that are unnecessary or unnecessary in a conversation.

[0121] "Removal" refers to the process of removing detected noise.

[0122] "Processed audio" refers to the clear audio after noise has been removed.

[0123] "Sending" refers to the process of delivering the generated processed voice to the other party.

[0124] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[0125] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[0126] "Generative AI model" refers to an algorithm or system that uses artificial intelligence to generate text or speech.

[0127] "New text" refers to the text generated by the generative AI model after noise has been removed.

[0128] "Listener" refers to a person who receives and listens to the distributed audio.

[0129] The system for carrying out this invention detects the characteristics of a user's speech, removes them as noise, generates processed speech, and transmits it. A specific embodiment of this system will be described below.

[0130] System configuration

[0131] This system uses the following hardware and software:

[0132] Hardware: Microphone, Smartphone

[0133] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (Hugging Face GPT-3 model)

[0134] Processing flow

[0135] 1. Speech recognition: The user's speech through the microphone is converted into text using the SpeechRecognition library, which allows the user's speech characteristics to be obtained as text data.

[0136] 2. Noise Removal: Remove noise phrases such as "um" and "well then" from the acquired text data. This process is done by detecting specific phrases and removing them from the text.

[0137] 3. Generative AI model: Based on the noise-removed text data, a generative AI model (GPT-3) is used to generate new text. This new text will have clear content with noise removed.

[0138] 4. Generate processed speech: The new text is converted into audio and generated as processed speech, which is a clearer speech with noise removed.

[0139] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive clear audio.

[0140] Specific examples

[0141] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide the stream to listeners as, "Today I'm going to talk about my new project."

[0142] Prompt Sentence Examples

[0143] Below are some example prompts to input to the generative AI model:

[0144] Remove noises like "um" and "well then" from user conversations to produce clearer voices.

[0145] By inputting this prompt into a generative AI model, clear audio with noise removed can be generated.

[0146] In this way, the present invention removes conversation noise in real time during live streaming or audio distribution, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[0147] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0148] Step 1:

[0149] The device collects the user's voice as they speak through a microphone. The device then converts the collected voice data into text data using the SpeechRecognition library. This process converts the voice data into text data. The input is voice data, and the output is text data.

[0150] Step 2:

[0151] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes any matching phrases. This process results in clean text data with the noise removed. The input is text data containing noise, and the output is text data with the noise removed.

[0152] Step 3:

[0153] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, it uses prompt sentences to give instructions to the generative AI model, which then generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new text data that has been generated.

[0154] Step 4:

[0155] The device inputs the generated new text data into a speech synthesis engine to generate processed speech. Specifically, the text data is passed to the speech synthesis engine, which outputs it as speech data. This process results in clear speech data with noise removed. The input is the generated new text data, and the output is processed speech data.

[0156] Step 5:

[0157] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[0158] Example 2

[0159] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] In conventional systems, the user's characteristic conversational habits and phrases remain intact, which can compromise anonymity. Furthermore, when collecting user opinions as text, manual conversion work is required, which is inefficient. This makes it difficult to completely protect the user's privacy when service personnel anonymously ask employees for user opinions.

[0161] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, means for generating processed voice that does not include the removed characteristics, means for converting the processed voice into text, and means for transmitting the text. This makes it possible to remove characteristic habits and phrases of the user and efficiently collect user opinions as text while maintaining anonymity.

[0162] "User conversation characteristics" refers to audio characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[0163] "Noise" refers to unnecessary features of a user's conversation that may compromise anonymity.

[0164] "Processed voice" refers to voice data generated by removing noise characteristic of a user's conversation and using the remaining content.

[0165] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[0166] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0167] "Means for converting to text" refers to the technology or system for analyzing audio data and converting its content into text information.

[0168] "Transmitting means" refers to the technology or system used to transmit the generated text or audio data to other devices or systems.

[0169] A "service representative" is a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[0170] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[0171] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[0172] MODE FOR CARRYING OUT THE INVENTION

[0173] The present invention provides a system for detecting characteristics of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[0174] Hardware and software used

[0175] Hardware: Servers, devices (PCs, smartphones, etc.)

[0176] Software: Generative AI models (e.g., GPT-4®), audio processing software (e.g., Audacity, Adobe Audition)

[0177] System Overview

[0178] The user uses a device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server then passes the received audio file through voice processing software, which removes the user's distinctive mannerisms and phrases. The removed audio is then generated as processed voice.

[0179] Next, the server inputs a prompt sentence to the generative AI model to convert the processed speech into text. For example, the prompt sentence can be "Please convert the following speech into text: [processed speech]." The generative AI model converts the speech to text based on the prompt sentence and returns the result to the server. The server then sends the converted text to the service representative. The service representative reviews the received text and anonymously understands the user's opinion.

[0180] Specific examples

[0181] Consider a scenario where a user says, "Well, then, I don't think product A is of good quality." When the user speaks into the device's microphone, the device records this speech and saves it as a digital audio file in WAV or MP3 format. The file name will include the date and time, such as "user_input_20231010.wav."

[0182] The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload." The server passes the received audio file to its voice processing software, which removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of Product A is good."

[0183] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates text saying, "I think the quality of product A is not good," and returns it to the server. The server then sends this text to the service representative via email or chat app. The transmitted content is, "User's opinion: I think the quality of product A is not good."

[0184] The service representative reviews the received text and anonymously captures the user's opinion, which removes the user's characteristic habits and phrases.

[0185] This system allows service personnel to collect user opinions anonymously and prevents users' characteristic habits and phrases from compromising that anonymity.

[0186] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0187] Step 1:

[0188] The user provides voice input. The user speaks into the device's microphone. For example, the user might say, "Well, then, I think the quality of product A is not good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[0189] Step 2:

[0190] The device records the audio and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's audio data, and the output is the saved digital audio file.

[0191] Step 3:

[0192] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The input is a digital audio file, and the output is the audio file sent to the server.

[0193] Step 4:

[0194] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server then passes the received audio file to audio processing software (e.g., Audacity's script function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[0195] Step 5:

[0196] The server inputs a prompt to the generative AI model to convert the processed speech into text. The server inputs the prompt to the generative AI model (e.g., GPT-4) as "Please convert the following speech into text: [processed speech]". The processed speech is attached to the prompt as binary data. The input is the processed speech and the prompt, and the output is the prompt sent to the generative AI model.

[0197] Step 6:

[0198] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and processed speech, and the output is the generated text.

[0199] Step 7:

[0200] The server sends the converted text to the service representative. The server then sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[0201] Step 8:

[0202] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This allows for opinions that have been removed of the user's characteristic habits and phrases. The input is the text sent to the service representative, and the output is the confirmed user opinion.

[0203] (Application example 2)

[0204] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] Conventional opinion collection systems leave users' characteristic habits and phrases intact, which can compromise anonymity. It is also difficult to collect opinions in real time, making it difficult to efficiently gather feedback. This makes it difficult for users to provide honest opinions, and there is the issue of how difficult it is to obtain the information needed to improve the service.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0207] In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed speech that does not include the removed features, thereby enabling efficient collection of opinions in real time while maintaining user anonymity.

[0208] "User" means any person or entity that uses the System to provide input.

[0209] "Conversational features" are individually identifiable elements such as habits, phrases, and intonation contained in a user's speech.

[0210] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[0211] "Removal" refers to the process of removing detected noise.

[0212] "Processed audio" is audio data from which noise has been removed.

[0213] "Send" is the act of transferring the processed audio to a designated recipient.

[0214] A "system" is a set of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[0215] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can also connect to the Internet and run applications.

[0216] An "application" is a software program that runs on a smartphone.

[0217] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[0218] "Real-time" refers to data processing and analysis occurring immediately.

[0219] "Security services" are services aimed at protecting information and ensuring anonymity.

[0220] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[0221] The system for implementing this invention executes a series of processes: detecting features of a user's speech, removing them as noise, generating processed speech, and transmitting it. Specifically, this is realized using an application installed on a smartphone.

[0222] Hardware and software used

[0223] Hardware: Smartphone

[0224] Software: Python, speech_recognition library, gTTS library

[0225] System Operation

[0226] 1. Speech Recognition: An application installed on a smartphone collects the user's speech, which is then converted into text using the speech_recognition library. This process uses a generative AI model to analyze the user's speech in real time.

[0227] 2. Text processing: The converted text is then stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "Well then," etc.).

[0228] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, which is a modified voice that removes the user's characteristic mannerisms and phrases.

[0229] 4. Transmission: The generated processed voice is transmitted to security service personnel, allowing them to collect opinions while maintaining the user's anonymity.

[0230] Specific examples

[0231] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[0232] Prompt Sentence Examples

[0233] Example prompts to input to a generative AI model:

[0234] Convert your users' speech to text and remove quirks and phrases to create an anonymous audio recording.

[0235] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[0236] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0237] Step 1:

[0238] The user launches the application installed on their smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[0239] Step 2:

[0240] The device converts collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze user speech in real time. Input: Collected voice data. Output: Converted text data.

[0241] Step 3:

[0242] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[0243] Step 4:

[0244] The device uses the gTTS library to convert the noise-removed text data back into speech. This speech is a processed speech that removes the user's characteristic habits and phrases. Input: Text data with noise removed. Output: Processed speech data.

[0245] Step 5:

[0246] The device sends the generated processed voice data to the security service staff. This allows the staff to collect opinions while maintaining the anonymity of the user. Input: Processed voice data. Output: Processed voice data sent to the staff.

[0247] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0248] "Example 1"

[0249] As one embodiment of the present invention, a system incorporating an emotion engine is provided. The system detects features of a user's speech, removes the features as noise, generates processed speech without the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the feature removal based on the emotions. For example, if the user is perceived as angry, the emotion engine may emphasize the angry features. Also, if the user is perceived as happy, the emotion engine may emphasize the happy features. This allows the user's emotions to be conveyed more accurately.

[0250] "Example 2"

[0251] The emotion engine also adjusts the generation of processed voices based on the user's emotion. For example, if the user feels sad, the emotion engine may emphasize the sadness characteristics. If the user feels surprised, the emotion engine may emphasize the surprise characteristics. This allows the user's emotion to be conveyed more accurately.

[0252] The processing flow of each embodiment will be described below.

[0253] "Example 1"

[0254] Step 1: Input the user's conversation into the emotion engine.

[0255] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0256] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[0257] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[0258] Step 5: Send the generated processed audio.

[0259] "Example 2"

[0260] Step 1: Input the user's conversation into the emotion engine.

[0261] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0262] Step 3: Adjust the generated processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[0263] Step 4: Generate processed audio based on the emotion-based adjustment results.

[0264] Step 5: Send the generated processed audio.

[0265] Example 1

[0266] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0267] Conventional voice analysis systems have had problems in effectively removing noises such as "umm" and "well then" that are included in user conversations, and also in not being able to properly reflect the user's emotions. As a result, the user's intentions and emotions are often not conveyed accurately.

[0268] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, means for generating processed voice that does not include the removed features, means for transmitting the processed voice, and means for analyzing the user's emotions and adjusting the removal of features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively removing noise from the user's conversation.

[0269] "User" refers to an individual or entity that uses the system to engage in conversations.

[0270] "Conversational features" refer to specific phrases and word patterns contained in a user's speech.

[0271] "Noise" refers to unnecessary phrases and words that disrupt the flow of conversation.

[0272] "Removal" refers to the process of removing detected noise from audio data.

[0273] "Processed audio" refers to audio data after noise has been removed.

[0274] "Transmit" refers to transferring the generated processed audio to another device or system.

[0275] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[0276] "Adjustment" refers to changing the noise removal method based on the results of sentiment analysis.

[0277] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases and words.

[0278] The present invention is a system that analyzes a user's conversation in real time, removes noise, and generates processed voice that reflects emotions. Specific embodiments of this system will be described below.

[0279] Hardware and software used

[0280] Hardware: Microphone, speaker, device (smartphone, tablet, PC)

[0281] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[0282] System configuration

[0283] Subject: Terminal

[0284] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[0285] Subject: Server

[0286] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects and removes noises such as "um" and "well then" from the text data. After the deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of removing the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, the "um" may not be removed to emphasize the anger.

[0287] The server converts the noise-removed text data back into audio data using text-to-speech synthesis technology, and the processed audio is sent from the server to the device.

[0288] Subject: Terminal

[0289] The device then plays the processed voice to the user through a speaker, allowing the user to hear the voice with noise removed and emotion reflected.

[0290] Specific examples

[0291] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. The emotion engine analyzes the user's emotions, and if it determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying, "Hello, it's a nice day today," is generated and sent to the device. The device then plays this processed voice back to the user.

[0292] Prompt Sentence Examples

[0293] "Detect characteristic phrases such as 'um' and 'well then' from the user's conversation and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, emphasize the angry features, and if the user is happy, emphasize the happy features."

[0294] The above is a specific embodiment for carrying out the present invention.

[0295] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0296] Step 1: Collecting audio data

[0297] Subject: Terminal

[0298] Input: User conversation

[0299] Specific operation: When the user starts speaking, the microphone on the device will automatically collect the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data will be recorded on the device.

[0300] Output: Collected audio data

[0301] Step 2: Sending audio data

[0302] Subject: Terminal

[0303] Input: Collected audio data

[0304] How it works: The device sends the collected voice data to the server in real time using an internet connection, such as Wi-Fi or mobile data.

[0305] Output: Audio data sent to the server

[0306] Step 3: Analyzing the audio data

[0307] Subject: Server

[0308] Input: Audio data sent to the server

[0309] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the generated text is "Hello, um, it's a nice day today."

[0310] Output: Text data

[0311] Step 4: Detect and remove noise

[0312] Subject: Server

[0313] Input: Text data

[0314] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" or "well then" from the text data. For example, the generated text would be "Hello, it's a nice day today."

[0315] Output: Text data with noise removed

[0316] Step 5: Sentiment analysis and adjustment

[0317] Subject: Server

[0318] Input: Text data with noise removed

[0319] Specific operation: The server analyzes the user's emotions using the emotion engine. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the server may not remove "um" to emphasize the angry characteristic.

[0320] Output: Text data reflecting emotions

[0321] Step 6: Generate processed audio

[0322] Subject: Server

[0323] Input: Text data that reflects emotions

[0324] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data such as "Hello, it's a nice day today" is generated.

[0325] Output: Processed audio data

[0326] Step 7: Send your processed audio

[0327] Subject: Server

[0328] Input: processed audio data

[0329] Specific operation: The server sends the generated processed audio to the device. This is done via an internet connection, such as Wi-Fi or mobile data.

[0330] Output: Processed audio data sent to the device

[0331] Step 8: Play the processed audio

[0332] Subject: Terminal

[0333] Input: Processed audio data sent to the device

[0334] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear "Hello, it's a nice day today."

[0335] Output: The processed audio played to the user

[0336] (Application example 1)

[0337] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] Conventional audio content distribution systems have the problem that unnecessary phrases and noise contained in the broadcaster's conversation are distributed as is, making the content difficult for listeners to hear. Also, since the broadcaster's emotions are not accurately conveyed, it is difficult to provide emotionally rich content to listeners.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0340] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed audio that does not include the deleted features, means for transmitting the processed audio, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed audio, and means for transmitting the generated processed audio. This makes it possible to delete unnecessary phrases from the distributor's conversation and provide listeners with audio content that emphasizes emotions in real time.

[0341] "User conversation characteristics" refers to specific phrases and word patterns included when a user speaks.

[0342] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[0343] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[0344] "Generative AI" refers to a system that uses artificial intelligence technology to generate text or speech.

[0345] An "emotion engine" is a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[0346] "Audio Content Provider" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[0347] A "listener" refers to a user who receives, watches, or listens to audio content.

[0348] "Real-time" refers to processing occurring simultaneously as the user speaks.

[0349] "Adjusting feature removal" refers to dynamically changing the selection of phrases and words to be removed based on the user's sentiment.

[0350] "Adjusted processed audio" refers to audio data after emotions have been emphasized and noise removed by the emotion engine.

[0351] A system for implementing this invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed speech that emphasizes emotions. A specific embodiment of this system will be described below.

[0352] System Configuration

[0353] The system consists of the following main components:

[0354] 1. Speech Recognition Module: Converts the user's speech into text.

[0355] 2. Generative AI model: Analyzes text and removes unnecessary phrases.

[0356] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[0357] 4. Speech generation module: converts the adjusted text into speech.

[0358] 5. Communication module: Transmits the generated processed audio to the listener.

[0359] Hardware and software used

[0360] Speech Recognition Module: Uses the speech_recognition library.

[0361] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[0362] Sentiment analysis engine: Uses the transformers library pipeline to utilize sentiment analysis models.

[0363] Speech generation module: Uses a TTS (Text-to-Speech) engine to convert text into speech.

[0364] Communications module: uses a communications protocol to transmit audio data over the Internet.

[0365] Data processing and calculation

[0366] 1. Speech recognition: Obtain the user's voice data and convert it to text using the speech_recognition library.

[0367] 2. Noise removal: Using a generative AI model to remove unnecessary phrases from the text.

[0368] 3. Sentiment analysis: Using a sentiment analysis engine, we analyze user sentiment from text.

[0369] 4. Sentiment adjustment: Adjust the text based on the analyzed sentiment.

[0370] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[0371] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[0372] Specific examples

[0373] For example, if a streamer says, "Well, today I'm going to talk about a new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about a new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[0374] Prompt Sentence Examples

[0375] User Spoken Text: "Well, today I'm going to talk about a new project. So, let me start by giving you an overview."

[0376] Sentiment analysis result: "Joy"

[0377] Generated text: "Today we'll be talking about a new project. Let's start with an overview."

[0378] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[0379] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0380] Step 1:

[0381] A user speaks into a microphone to deliver audio content. The device captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[0382] Step 2:

[0383] The device sends the acquired voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data into text. The input is the voice data, and the output is the converted text data.

[0384] Step 3:

[0385] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[0386] Step 4:

[0387] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[0388] Step 5:

[0389] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, it processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with emphasized emotions.

[0390] Step 6:

[0391] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[0392] Step 7:

[0393] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data via the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[0394] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and emotionally-emphasized voice content can be provided to the listener.

[0395] Example 2

[0396] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] Conventional systems have difficulty not only removing the characteristics of a user's speech but also generating speech that accurately reflects the user's emotions. Furthermore, when collecting user opinions anonymously, there is a risk that the user's distinctive habits and phrases may remain, compromising anonymity. Furthermore, it is difficult for service personnel to accurately understand the user's emotions.

[0398] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0399] In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, and means for generating processed speech that does not include the removed characteristics. This makes it possible to remove the user's characteristic habits and phrases and collect user opinions while maintaining anonymity. The server also includes means for analyzing the user's emotions and means for generating processed speech that reflects the analyzed emotions. This allows the generated speech to accurately reflect the user's emotions, enabling the service representative to accurately understand the user's emotions.

[0400] "User conversation characteristics" refers to speech characteristics such as habits, phrases, and intonation that are specific to the user's speech.

[0401] "Noise" refers to distinctive habits and phrases in a user's conversations that may compromise anonymity.

[0402] "Processed voice" refers to voice data in which the characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[0403] "Means for analyzing emotions" refers to technologies and algorithms for detecting emotions from users' text data and voice data, and classifying and evaluating those emotions.

[0404] "Means for generating processed voice that reflects emotions" refers to technologies and algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions, and generating voice data that reflects the user's emotions.

[0405] A "service representative" is a person whose role is to collect user opinions and feedback and to improve or respond to the service based on that information.

[0406] "Anonymity" refers to a state in which a user's personal information or specific characteristics are protected from being known to third parties.

[0407] MODE FOR CARRYING OUT THE INVENTION

[0408] The present invention relates to a system for generating processed speech that reflects emotions by removing characteristics of a user's speech. Specific embodiments of this system will be described below.

[0409] System Overview

[0410] The system receives the user's voice data and converts it into text using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data, analyzes the user's emotions using an emotion engine, and finally uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[0411] Hardware and software used

[0412] Speech recognition software: Google Cloud Speech-to-Text API

[0413] Natural language processing libraries: spaCy, NLTK

[0414] Emotion Engine: IBM Watson(R) Tone Analyzer

[0415] Text-to-speech software: Amazon Polly

[0416] Specific examples

[0417] Consider the case where a user says, "Well, then, I don't think the quality of product A is good." The system works as follows.

[0418] 1. User voice input: The user speaks into the smartphone microphone, "Well, then, I don't think the quality of product A is good."

[0419] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[0420] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the speech data into text data such as, "Well, then, I don't think the quality of product A is good."

[0421] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Well, then," from the text data, generating text data such as "I don't think the quality of product A is good."

[0422] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[0423] 6. Generate processed speech: The server uses Amazon Polly to convert the text data, "I think the quality of product A is poor," into processed speech with a tone that reflects sadness.

[0424] 7. Sending the processed voice: The server sends the generated processed voice to the service representative, who can anonymously listen to the user's opinion through this processed voice.

[0425] Prompt Sentence Examples

[0426] An example of a prompt to input to a generative AI model is as follows:

[0427] User says: "Well, then, I don't think product A is of good quality."

[0428] Prompt for generative AI model: "Remove the user's utterances and generate a modified voice that reflects their emotions."

[0429] In this way, service personnel can listen to users' opinions anonymously and prevent their anonymity from being compromised by their characteristic habits or phrases. Furthermore, by processing the voice to reflect their emotions, service personnel can accurately understand the user's feelings.

[0430] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0431] Step 1:

[0432] The user inputs speech.

[0433] A user inputs voice using a device (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the device in a digital format (e.g., WAV or MP3).

[0434] Step 2:

[0435] The server receives the audio data.

[0436] The server receives the voice data sent from the device. The received voice data is stored in the server's storage. The input is the voice data from the device, and the output is the voice data stored in the server.

[0437] Step 3:

[0438] The server converts the voice data into text data.

[0439] The server uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[0440] Step 4:

[0441] The server removes characteristic habits and phrases from the text data.

[0442] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data with the characteristic habits and phrases removed. Specifically, unnecessary phrases such as "Well, then," are removed, and the text data that reads "I don't think the quality of product A is good" is generated.

[0443] Step 5:

[0444] The server analyzes the sentiment of the text data.

[0445] The server uses an emotion engine (for example, IBM Watson Tone Analyzer) to analyze the user's emotions from the text data. The input is text data from which characteristic habits and phrases have been removed, and the output is analyzed emotion data. Specifically, emotions such as "sad" are analyzed from the text data.

[0446] Step 6:

[0447] The server generates processed voice that reflects the emotion.

[0448] The server uses speech synthesis software (e.g., Amazon Polly) to generate processed speech that reflects emotions. The input is the analyzed emotion data and text data, and the output is processed speech that reflects emotions. Specifically, the text data, "I think the quality of product A is not good," is converted into processed speech with a tone that reflects sadness.

[0449] Step 7:

[0450] The server sends the processed voice to the service representative.

[0451] The server sends the generated processed voice to the service representative. The input is the processed voice that reflects the emotion, and the output is the processed voice sent to the service representative. The service representative can listen to the user's opinion anonymously through this processed voice.

[0452] (Application example 2)

[0453] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0454] Conventional anonymous reporting systems have had problems such as the possibility that a user's distinctive habits or phrases could compromise anonymity, and it is difficult to accurately convey the user's feelings. This can lead to the credibility and urgency of the report not being properly assessed.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0456] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for detecting the user's emotions and adjusting the processed voice based on the emotions, and means for transmitting the adjusted processed voice, thereby enabling anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[0457] "Characteristics of a user's conversation" refers to characteristics and habits of the user's voice when speaking, specific phrases, and the like.

[0458] "Removing noise" refers to a process of removing the characteristics of a user's speech from the voice data.

[0459] "Processed voice" refers to voice data generated after removing features of the user's speech.

[0460] "Means for transmitting" refers to a function for transmitting the generated processed voice to another device or server.

[0461] "Detecting emotions" refers to the process of analyzing emotions from the content of a user's speech and the tone of their voice.

[0462] "Adjusting the processed voice based on emotion" refers to processing voice data to reflect the detected emotion.

[0463] "Making an anonymous report" refers to the act of sending a report while concealing the user's personal information and characteristics.

[0464] A system for implementing this invention detects characteristics of a user's speech, deletes the characteristics as noise, generates processed speech that does not include the deleted characteristics, detects the user's emotions, adjusts the processed speech based on the emotions, and finally transmits the adjusted processed speech.

[0465] Hardware and software used

[0466] Hardware: Smartphone

[0467] Software: Python, speech_recognition library, pydub library, librosa library, soundfile library

[0468] Processing flow

[0469] 1. Speech recognition: The user's voice is recorded using the smartphone's microphone, and text is extracted from the voice data using the speech_recognition library. This allows the user's speech to be obtained as text data.

[0470] 2. Voice processing: Using the librosa library, we remove the user's characteristic habits and phrases from the voice data. Specifically, we analyze the characteristics of the voice and remove the characteristic parts as noise.

[0471] 3. Emotion detection: Using the librosa library, the tone and pitch of the voice data are analyzed to detect the user's emotion. For example, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0472] 4. Emotion-based audio adjustment: Reprocessing audio data based on detected emotions, for example slowing down the audio to emphasize sadness or raising the pitch to emphasize surprise.

[0473] 5. Audio transmission: The processed audio data is saved using the soundfile library and sent to the reporting center using the smartphone's communication function.

[0474] Specific examples

[0475] If a user reports, "I saw a suspicious person in my neighborhood," the system generates a processed voice saying, "I saw a suspicious person," and sends it to the reporting center. Furthermore, if the user is feeling anxious, the system can emphasize that feeling.

[0476] Prompt Sentence Examples

[0477] When a user reports "I saw a suspicious person in my neighborhood," the application generates a processed voice saying "I saw a suspicious person" and sends it to the reporting center. Furthermore, if the user is feeling anxious, the application can emphasize the emotion of anxiety.

[0478] In this way, anonymous reporting that accurately conveys emotions is possible while maintaining the anonymity of the user.

[0479] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0480] Step 1:

[0481] The user records audio using the smartphone microphone. The input is the user's speech, and the output is an audio file (e.g., input.wav). This audio file is used in subsequent processing steps.

[0482] Step 2:

[0483] The device uses the speech_recognition library to extract text from an audio file. The input is an audio file, and the output is text data (e.g., "I saw a suspicious person in my neighborhood"). This text data is used to capture the user's spoken content as text information.

[0484] Step 3:

[0485] The device uses the librosa library to remove the user's characteristic habits and phrases from the voice data. The input is an audio file, and the output is the voice data with the characteristics removed. Specifically, the device analyzes the characteristics of the voice and removes the characteristic parts as noise.

[0486] Step 4:

[0487] The device uses the librosa library to analyze the tone and pitch of the voice data to detect the user's emotions. The input is an audio file, and the output is the detected emotion (e.g., "anxiety"). Specifically, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0488] Step 5:

[0489] The device reprocesses the audio data based on the detected emotion. The input is the feature-removed audio data and the detected emotion, and the output is processed audio with the emotion emphasized. For example, the audio may be slowed down to emphasize sadness, or the pitch may be raised to emphasize surprise.

[0490] Step 6:

[0491] The device uses the soundfile library to save the processed audio data. The input is the processed audio with enhanced emotion, and the output is the saved audio file (e.g., output.wav). This audio file is used to send to the reporting center.

[0492] Step 7:

[0493] The device uses the smartphone's communication function to send the processed audio file to the reporting center. The input is the saved audio file, and the output is the audio data sent to the reporting center. This allows anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[0494] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0495] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0496] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0497] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0498] [Second embodiment]

[0499] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0500] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0501] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0502] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0503] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0504] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0505] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0506] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0507] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0508] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0509] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0510] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0511] "Example 1"

[0512] One aspect of the present invention is a system that uses a generation AI to analyze a user's conversation in real time. This system detects characteristics of the user's conversation, such as phrases like "um" and "well then," and removes them as noise. The removal occurs the moment the generation AI detects a characteristic phrase, resulting in the generation of processed speech that does not contain the characteristic phrase. The generated processed speech is then transmitted to the other party through the system.

[0513] "Example 2"

[0514] As a specific example, a service representative may want to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the user's speech characteristics. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask for the user's opinion without losing their anonymity due to the user's distinctive mannerisms or phrases.

[0515] The processing flow of each embodiment will be described below.

[0516] "Example 1"

[0517] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[0518] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[0519] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[0520] Step 4: A processed voice without any features is generated and sent to the other party through the system.

[0521] "Example 2"

[0522] Step 1: When a service representative wants to anonymously ask a user for their opinion, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[0523] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[0524] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[0525] Step 4: A processed voice without the user's characteristics is generated and sent to the service representative. This allows the service representative to listen to the user's opinions anonymously and prevents the user's characteristic habits and phrases from compromising the anonymity.

[0526] Example 1

[0527] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0528] Conventional voice communication systems have a problem in that characteristic phrases such as "umm" and "well then" contained in user conversations remain as noise, making the voice difficult for the other party to hear. In addition, manually removing these noises requires a great deal of effort, making it impractical for real-time conversations. Furthermore, even when collecting user opinions anonymously, the presence of noise makes it difficult to accurately grasp their opinions.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0530] In this invention, the server includes means for detecting features of a user's speech, means for removing the features as noise, and means for generating processed speech that does not include the removed features, thereby making it possible to remove noise from a user's speech in real time and generate clear speech.

[0531] "User" refers to an individual or entity that uses the system to engage in conversations.

[0532] "Conversational features" refer to specific phrases or words such as "um" or "well then" that are included in the user's speech.

[0533] "Noise" refers to unnecessary phrases and words that are detected as characteristics of conversation and are targeted for removal.

[0534] "Processed audio" refers to clear audio data after noise has been removed.

[0535] "Server" refers to a computer system for processing and analyzing voice data.

[0536] "Terminal" refers to a device used by a user that records and plays audio.

[0537] "Speech recognition software" refers to a program for converting voice data into text data.

[0538] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[0539] "Text data" refers to textual information converted by voice recognition software.

[0540] "Clean text data" refers to text data after noise has been removed.

[0541] "Voice data" refers to data that is a digital representation of a user's speech.

[0542] "Real-time" refers to processing occurring simultaneously with the user's conversation.

[0543] The present invention relates to a system for analyzing user conversations in real time and removing noise. Specific embodiments of this system will be described below.

[0544] System configuration

[0545] Hardware

[0546] Server: Uses a computer system equipped with high-performance CPUs and GPUs, which allows for large amounts of data processing and the execution of generative AI models.

[0547] Device: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[0548] software

[0549] Speech Recognition Software: We use common speech recognition APIs (e.g., Google Speech-to-Text API) to convert voice data into text data.

[0550] Generative AI models: Use generative AI models (e.g., OpenAI's GPT-3) to analyze user conversations and detect characteristic phrases.

[0551] System Operation

[0552] Acquiring and sending audio data

[0553] When a user starts talking, the device uses a built-in microphone to record the user's voice, and the recorded voice data is sent to the server in real time.

[0554] Converting audio data to text

[0555] The server uses speech recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted directly into text data.

[0556] Text data analysis and noise removal

[0557] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. These phrases are removed as noise, and clean text data is generated.

[0558] Speech conversion of clean text data

[0559] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[0560] Sending and playing processed audio

[0561] The generated processed voice is sent from the server to the terminal, which then plays it back, allowing the user to hear clearer voice with the noise removed.

[0562] Examples of specific examples and prompts

[0563] For example, suppose a user has the following conversation:

[0564] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0565] The server receives this conversation and converts it into the following text data using speech recognition software:

[0566] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0567] Next, we use a generative AI model to remove noise as follows:

[0568] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[0569] Finally, the server converts the cleaned text data back into audio data, generates processed audio, and sends it to the terminal, which plays the processed audio.

[0570] An example of a prompt sentence might be:

[0571] "Detect distinctive phrases in user conversations and remove them as noise, such as 'um' or 'Well then.'"

[0572] By inputting this prompt into a generative AI model, a process is performed to remove noise from the user's conversation.

[0573] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0574] Step 1:

[0575] A user initiates a conversation.

[0576] The user begins speaking to the system, for example, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[0577] Step 2:

[0578] The device records the user's voice and sends it to the server.

[0579] The device records the user's voice using a built-in microphone. The recorded voice data is sent to the server in real time. The input is the user's voice data, and the output is the voice data sent to the server.

[0580] Step 3:

[0581] The server uses voice recognition software to convert the voice data into text data.

[0582] The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[0583] Step 4:

[0584] The server uses the generative AI model to analyze the text data and detect characteristic phrases.

[0585] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[0586] Step 5:

[0587] The server removes the detected phrases as noise and generates clean text data.

[0588] The server removes the detected characteristic phrases as noise. For example, from the text data "Um, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Um" and "So, let's go for a walk in the afternoon" to generate clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[0589] Step 6:

[0590] The server reconverts the clean text data into audio data.

[0591] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk this afternoon" is generated. The input is cleaned text data, and the output is speech data.

[0592] Step 7:

[0593] The server transmits the generated processed voice to the terminal.

[0594] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[0595] Step 8:

[0596] The device plays the processed audio.

[0597] The device then plays back the processed audio, allowing the user to hear clear audio with the noise removed. The input is the audio data sent to the device, and the output is the audio played back from the device's speaker.

[0598] (Application example 1)

[0599] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0600] Conventional audio distribution systems have the problem that noises such as "umm" and "well then" that occur during a streamer's conversation are transmitted to the listener, degrading the quality of the audio. Streamers are also required to consciously speak to avoid the noise, making it difficult to have a natural conversation.

[0601] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed audio that does not include the removed features. This makes it possible to remove noise from conversation in real time during live streaming or audio distribution. Furthermore, by including means for generating new text based on text from which noise has been removed using a generative AI model, the broadcaster can provide clear audio to listeners while continuing a natural conversation without having to consciously avoid noise.

[0602] "User speech characteristics" refers to specific phrases and word patterns that a user uses when speaking.

[0603] "Noise" refers to phrases or words that are unnecessary or unnecessary in a conversation.

[0604] "Removal" refers to the process of removing detected noise.

[0605] "Processed audio" refers to the clear audio after noise has been removed.

[0606] "Sending" refers to the process of delivering the generated processed voice to the other party.

[0607] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[0608] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[0609] "Generative AI model" refers to an algorithm or system that uses artificial intelligence to generate text or speech.

[0610] "New text" refers to the text generated by the generative AI model after noise has been removed.

[0611] "Listener" refers to a person who receives and listens to the distributed audio.

[0612] The system for carrying out this invention detects the characteristics of a user's speech, removes them as noise, generates processed speech, and transmits it. A specific embodiment of this system will be described below.

[0613] System configuration

[0614] This system uses the following hardware and software:

[0615] Hardware: Microphone, Smartphone

[0616] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (Hugging Face GPT-3 model)

[0617] Processing flow

[0618] 1. Speech recognition: The user's speech through the microphone is converted into text using the SpeechRecognition library, which allows the user's speech characteristics to be obtained as text data.

[0619] 2. Noise Removal: Remove noise phrases such as "um" and "well then" from the acquired text data. This process is done by detecting specific phrases and removing them from the text.

[0620] 3. Generative AI model: Based on the noise-removed text data, a generative AI model (GPT-3) is used to generate new text. This new text will have clear content with noise removed.

[0621] 4. Generate processed speech: The new text is converted into audio and generated as processed speech, which is a clearer speech with noise removed.

[0622] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive clear audio.

[0623] Specific examples

[0624] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide the stream to listeners as, "Today I'm going to talk about my new project."

[0625] Prompt Sentence Examples

[0626] Below are some example prompts to input to the generative AI model:

[0627] Remove noises like "um" and "well then" from user conversations to produce clearer voices.

[0628] By inputting this prompt into a generative AI model, clear audio with noise removed can be generated.

[0629] In this way, the present invention removes conversation noise in real time during live streaming or audio distribution, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[0630] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0631] Step 1:

[0632] The device collects the user's voice as they speak through a microphone. The device then converts the collected voice data into text data using the SpeechRecognition library. This process converts the voice data into text data. The input is voice data, and the output is text data.

[0633] Step 2:

[0634] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes any matching phrases. This process results in clean text data with the noise removed. The input is text data containing noise, and the output is text data with the noise removed.

[0635] Step 3:

[0636] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, it uses prompt sentences to give instructions to the generative AI model, which then generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new text data that has been generated.

[0637] Step 4:

[0638] The device inputs the generated new text data into a speech synthesis engine to generate processed speech. Specifically, the text data is passed to the speech synthesis engine, which outputs it as speech data. This process results in clear speech data with noise removed. The input is the generated new text data, and the output is processed speech data.

[0639] Step 5:

[0640] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[0641] Example 2

[0642] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] In conventional systems, the user's characteristic conversational habits and phrases remain intact, which can compromise anonymity. Furthermore, when collecting user opinions as text, manual conversion work is required, which is inefficient. This makes it difficult to completely protect the user's privacy when service personnel anonymously ask employees for user opinions.

[0644] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, means for generating processed voice that does not include the removed characteristics, means for converting the processed voice into text, and means for transmitting the text. This makes it possible to remove characteristic habits and phrases of the user and efficiently collect user opinions as text while maintaining anonymity.

[0645] "User conversation characteristics" refers to audio characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[0646] "Noise" refers to unnecessary features of a user's conversation that may compromise anonymity.

[0647] "Processed voice" refers to voice data generated by removing noise characteristic of a user's conversation and using the remaining content.

[0648] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[0649] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[0650] "Means for converting to text" refers to the technology or system for analyzing audio data and converting its content into text information.

[0651] "Transmitting means" refers to the technology or system used to transmit the generated text or audio data to other devices or systems.

[0652] A "service representative" is a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[0653] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[0654] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[0655] MODE FOR CARRYING OUT THE INVENTION

[0656] The present invention provides a system for detecting characteristics of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[0657] Hardware and software used

[0658] Hardware: Servers, devices (PCs, smartphones, etc.)

[0659] Software: Generative AI models (e.g., GPT-4), audio processing software (e.g., Audacity, Adobe Audition)

[0660] System Overview

[0661] The user uses a device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server then passes the received audio file through voice processing software, which removes the user's distinctive mannerisms and phrases. The removed audio is then generated as processed voice.

[0662] Next, the server inputs a prompt sentence to the generative AI model to convert the processed speech into text. For example, the prompt sentence can be "Please convert the following speech into text: [processed speech]." The generative AI model converts the speech to text based on the prompt sentence and returns the result to the server. The server then sends the converted text to the service representative. The service representative reviews the received text and anonymously understands the user's opinion.

[0663] Specific examples

[0664] Consider a scenario where a user says, "Well, then, I don't think product A is of good quality." When the user speaks into the device's microphone, the device records this speech and saves it as a digital audio file in WAV or MP3 format. The file name will include the date and time, such as "user_input_20231010.wav."

[0665] The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload." The server passes the received audio file to its voice processing software, which removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of Product A is good."

[0666] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates text saying, "I think the quality of product A is not good," and returns it to the server. The server then sends this text to the service representative via email or chat app. The transmitted content is, "User's opinion: I think the quality of product A is not good."

[0667] The service representative reviews the received text and anonymously captures the user's opinion, which removes the user's characteristic habits and phrases.

[0668] This system allows service personnel to collect user opinions anonymously and prevents users' characteristic habits and phrases from compromising that anonymity.

[0669] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0670] Step 1:

[0671] The user provides voice input. The user speaks into the device's microphone. For example, the user might say, "Well, then, I think the quality of product A is not good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[0672] Step 2:

[0673] The device records the audio and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's audio data, and the output is the saved digital audio file.

[0674] Step 3:

[0675] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The input is a digital audio file, and the output is the audio file sent to the server.

[0676] Step 4:

[0677] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server then passes the received audio file to audio processing software (e.g., Audacity's script function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[0678] Step 5:

[0679] The server inputs a prompt to the generative AI model to convert the processed speech into text. The server inputs the prompt to the generative AI model (e.g., GPT-4) as "Please convert the following speech into text: [processed speech]". The processed speech is attached to the prompt as binary data. The input is the processed speech and the prompt, and the output is the prompt sent to the generative AI model.

[0680] Step 6:

[0681] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and processed speech, and the output is the generated text.

[0682] Step 7:

[0683] The server sends the converted text to the service representative. The server then sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[0684] Step 8:

[0685] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This allows for opinions that have been removed of the user's characteristic habits and phrases. The input is the text sent to the service representative, and the output is the confirmed user opinion.

[0686] (Application example 2)

[0687] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] Conventional opinion collection systems leave users' characteristic habits and phrases intact, which can compromise anonymity. It is also difficult to collect opinions in real time, making it difficult to efficiently gather feedback. This makes it difficult for users to provide honest opinions, and there is the issue of how difficult it is to obtain the information needed to improve the service.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0690] In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed speech that does not include the removed features, thereby enabling efficient collection of opinions in real time while maintaining user anonymity.

[0691] "User" means any person or entity that uses the System to provide input.

[0692] "Conversational features" are individually identifiable elements such as habits, phrases, and intonation contained in a user's speech.

[0693] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[0694] "Removal" refers to the process of removing detected noise.

[0695] "Processed audio" is audio data from which noise has been removed.

[0696] "Send" is the act of transferring the processed audio to a designated recipient.

[0697] A "system" is a set of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[0698] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can also connect to the Internet and run applications.

[0699] An "application" is a software program that runs on a smartphone.

[0700] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[0701] "Real-time" refers to data processing and analysis occurring immediately.

[0702] "Security services" are services aimed at protecting information and ensuring anonymity.

[0703] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[0704] The system for implementing this invention executes a series of processes: detecting features of a user's speech, removing them as noise, generating processed speech, and transmitting it. Specifically, this is realized using an application installed on a smartphone.

[0705] Hardware and software used

[0706] Hardware: Smartphone

[0707] Software: Python, speech_recognition library, gTTS library

[0708] System Operation

[0709] 1. Speech Recognition: An application installed on a smartphone collects the user's speech, which is then converted into text using the speech_recognition library. This process uses a generative AI model to analyze the user's speech in real time.

[0710] 2. Text processing: The converted text is then stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "Well then," etc.).

[0711] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, which is a modified voice that removes the user's characteristic mannerisms and phrases.

[0712] 4. Transmission: The generated processed voice is transmitted to security service personnel, allowing them to collect opinions while maintaining the user's anonymity.

[0713] Specific examples

[0714] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[0715] Prompt Sentence Examples

[0716] Example prompts to input to a generative AI model:

[0717] Convert your users' speech to text and remove quirks and phrases to create an anonymous audio recording.

[0718] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[0719] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0720] Step 1:

[0721] The user launches the application installed on their smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[0722] Step 2:

[0723] The device converts collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze user speech in real time. Input: Collected voice data. Output: Converted text data.

[0724] Step 3:

[0725] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[0726] Step 4:

[0727] The device uses the gTTS library to convert the noise-removed text data back into speech. This speech is a processed speech that removes the user's characteristic habits and phrases. Input: Text data with noise removed. Output: Processed speech data.

[0728] Step 5:

[0729] The device sends the generated processed voice data to the security service staff. This allows the staff to collect opinions while maintaining the anonymity of the user. Input: Processed voice data. Output: Processed voice data sent to the staff.

[0730] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0731] "Example 1"

[0732] As one embodiment of the present invention, a system incorporating an emotion engine is provided. The system detects features of a user's speech, removes the features as noise, generates processed speech without the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the feature removal based on the emotions. For example, if the user is perceived as angry, the emotion engine may emphasize the angry features. Also, if the user is perceived as happy, the emotion engine may emphasize the happy features. This allows the user's emotions to be conveyed more accurately.

[0733] "Example 2"

[0734] The emotion engine also adjusts the generation of processed voices based on the user's emotion. For example, if the user feels sad, the emotion engine may emphasize the sadness characteristics. If the user feels surprised, the emotion engine may emphasize the surprise characteristics. This allows the user's emotion to be conveyed more accurately.

[0735] The processing flow of each embodiment will be described below.

[0736] "Example 1"

[0737] Step 1: Input the user's conversation into the emotion engine.

[0738] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0739] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[0740] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[0741] Step 5: Send the generated processed audio.

[0742] "Example 2"

[0743] Step 1: Input the user's conversation into the emotion engine.

[0744] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0745] Step 3: Adjust the generated processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[0746] Step 4: Generate processed audio based on the emotion-based adjustment results.

[0747] Step 5: Send the generated processed audio.

[0748] Example 1

[0749] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] Conventional voice analysis systems have had problems in effectively removing noises such as "umm" and "well then" that are included in user conversations, and also in not being able to properly reflect the user's emotions. As a result, the user's intentions and emotions are often not conveyed accurately.

[0751] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, means for generating processed voice that does not include the removed features, means for transmitting the processed voice, and means for analyzing the user's emotions and adjusting the removal of features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively removing noise from the user's conversation.

[0752] "User" refers to an individual or entity that uses the system to engage in conversations.

[0753] "Conversational features" refer to specific phrases and word patterns contained in a user's speech.

[0754] "Noise" refers to unnecessary phrases and words that disrupt the flow of conversation.

[0755] "Removal" refers to the process of removing detected noise from audio data.

[0756] "Processed audio" refers to audio data after noise has been removed.

[0757] "Transmit" refers to transferring the generated processed audio to another device or system.

[0758] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[0759] "Adjustment" refers to changing the noise removal method based on the results of sentiment analysis.

[0760] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases and words.

[0761] The present invention is a system that analyzes a user's conversation in real time, removes noise, and generates processed voice that reflects emotions. Specific embodiments of this system will be described below.

[0762] Hardware and software used

[0763] Hardware: Microphone, speaker, device (smartphone, tablet, PC)

[0764] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[0765] System configuration

[0766] Subject: Terminal

[0767] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[0768] Subject: Server

[0769] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects and removes noises such as "um" and "well then" from the text data. After the deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of removing the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, the "um" may not be removed to emphasize the anger.

[0770] The server converts the noise-removed text data back into audio data using text-to-speech synthesis technology, and the processed audio is sent from the server to the device.

[0771] Subject: Terminal

[0772] The device then plays the processed voice to the user through a speaker, allowing the user to hear the voice with noise removed and emotion reflected.

[0773] Specific examples

[0774] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. The emotion engine analyzes the user's emotions, and if it determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying, "Hello, it's a nice day today," is generated and sent to the device. The device then plays this processed voice back to the user.

[0775] Prompt Sentence Examples

[0776] "Detect characteristic phrases such as 'um' and 'well then' from the user's conversation and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, emphasize the angry features, and if the user is happy, emphasize the happy features."

[0777] The above is a specific embodiment for carrying out the present invention.

[0778] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0779] Step 1: Collecting audio data

[0780] Subject: Terminal

[0781] Input: User conversation

[0782] Specific operation: When the user starts speaking, the microphone on the device will automatically collect the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data will be recorded on the device.

[0783] Output: Collected audio data

[0784] Step 2: Sending audio data

[0785] Subject: Terminal

[0786] Input: Collected audio data

[0787] How it works: The device sends the collected voice data to the server in real time using an internet connection, such as Wi-Fi or mobile data.

[0788] Output: Audio data sent to the server

[0789] Step 3: Analyzing the audio data

[0790] Subject: Server

[0791] Input: Audio data sent to the server

[0792] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the generated text is "Hello, um, it's a nice day today."

[0793] Output: Text data

[0794] Step 4: Detect and remove noise

[0795] Subject: Server

[0796] Input: Text data

[0797] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" or "well then" from the text data. For example, the generated text would be "Hello, it's a nice day today."

[0798] Output: Text data with noise removed

[0799] Step 5: Sentiment analysis and adjustment

[0800] Subject: Server

[0801] Input: Text data with noise removed

[0802] Specific operation: The server analyzes the user's emotions using the emotion engine. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the server may not remove "um" to emphasize the angry characteristic.

[0803] Output: Text data reflecting emotions

[0804] Step 6: Generate processed audio

[0805] Subject: Server

[0806] Input: Text data that reflects emotions

[0807] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data such as "Hello, it's a nice day today" is generated.

[0808] Output: Processed audio data

[0809] Step 7: Send your processed audio

[0810] Subject: Server

[0811] Input: processed audio data

[0812] Specific operation: The server sends the generated processed audio to the device. This is done via an internet connection, such as Wi-Fi or mobile data.

[0813] Output: Processed audio data sent to the device

[0814] Step 8: Play the processed audio

[0815] Subject: Terminal

[0816] Input: Processed audio data sent to the device

[0817] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear "Hello, it's a nice day today."

[0818] Output: The processed audio played to the user

[0819] (Application example 1)

[0820] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0821] Conventional audio content distribution systems have the problem that unnecessary phrases and noise contained in the broadcaster's conversation are distributed as is, making the content difficult for listeners to hear. Also, since the broadcaster's emotions are not accurately conveyed, it is difficult to provide emotionally rich content to listeners.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0823] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed audio that does not include the deleted features, means for transmitting the processed audio, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed audio, and means for transmitting the generated processed audio. This makes it possible to delete unnecessary phrases from the distributor's conversation and provide listeners with audio content that emphasizes emotions in real time.

[0824] "User conversation characteristics" refers to specific phrases and word patterns included when a user speaks.

[0825] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[0826] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[0827] "Generative AI" refers to a system that uses artificial intelligence technology to generate text or speech.

[0828] An "emotion engine" is a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[0829] "Audio Content Provider" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[0830] A "listener" refers to a user who receives, watches, or listens to audio content.

[0831] "Real-time" refers to processing occurring simultaneously as the user speaks.

[0832] "Adjusting feature removal" refers to dynamically changing the selection of phrases and words to be removed based on the user's sentiment.

[0833] "Adjusted processed audio" refers to audio data after emotions have been emphasized and noise removed by the emotion engine.

[0834] A system for implementing this invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed speech that emphasizes emotions. A specific embodiment of this system will be described below.

[0835] System Configuration

[0836] The system consists of the following main components:

[0837] 1. Speech Recognition Module: Converts the user's speech into text.

[0838] 2. Generative AI model: Analyzes text and removes unnecessary phrases.

[0839] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[0840] 4. Speech generation module: converts the adjusted text into speech.

[0841] 5. Communication module: Transmits the generated processed audio to the listener.

[0842] Hardware and software used

[0843] Speech Recognition Module: Uses the speech_recognition library.

[0844] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[0845] Sentiment analysis engine: Uses the transformers library pipeline to utilize sentiment analysis models.

[0846] Speech generation module: Uses a TTS (Text-to-Speech) engine to convert text into speech.

[0847] Communications module: uses a communications protocol to transmit audio data over the Internet.

[0848] Data processing and calculation

[0849] 1. Speech recognition: Obtain the user's voice data and convert it to text using the speech_recognition library.

[0850] 2. Noise removal: Using a generative AI model to remove unnecessary phrases from the text.

[0851] 3. Sentiment analysis: Using a sentiment analysis engine, we analyze user sentiment from text.

[0852] 4. Sentiment adjustment: Adjust the text based on the analyzed sentiment.

[0853] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[0854] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[0855] Specific examples

[0856] For example, if a streamer says, "Well, today I'm going to talk about a new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about a new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[0857] Prompt Sentence Examples

[0858] User Spoken Text: "Well, today I'm going to talk about a new project. So, let me start by giving you an overview."

[0859] Sentiment analysis result: "Joy"

[0860] Generated text: "Today we'll be talking about a new project. Let's start with an overview."

[0861] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[0862] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0863] Step 1:

[0864] A user speaks into a microphone to deliver audio content. The device captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[0865] Step 2:

[0866] The device sends the acquired voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data into text. The input is the voice data, and the output is the converted text data.

[0867] Step 3:

[0868] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[0869] Step 4:

[0870] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[0871] Step 5:

[0872] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, it processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with emphasized emotions.

[0873] Step 6:

[0874] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[0875] Step 7:

[0876] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data via the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[0877] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and emotionally-emphasized voice content can be provided to the listener.

[0878] Example 2

[0879] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0880] Conventional systems have difficulty not only removing the characteristics of a user's speech but also generating speech that accurately reflects the user's emotions. Furthermore, when collecting user opinions anonymously, there is a risk that the user's distinctive habits and phrases may remain, compromising anonymity. Furthermore, it is difficult for service personnel to accurately understand the user's emotions.

[0881] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0882] In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, and means for generating processed speech that does not include the removed characteristics. This makes it possible to remove the user's characteristic habits and phrases and collect user opinions while maintaining anonymity. The server also includes means for analyzing the user's emotions and means for generating processed speech that reflects the analyzed emotions. This allows the generated speech to accurately reflect the user's emotions, enabling the service representative to accurately understand the user's emotions.

[0883] "User conversation characteristics" refers to speech characteristics such as habits, phrases, and intonation that are specific to the user's speech.

[0884] "Noise" refers to distinctive habits and phrases in a user's conversations that may compromise anonymity.

[0885] "Processed voice" refers to voice data in which the characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[0886] "Means for analyzing emotions" refers to technologies and algorithms for detecting emotions from users' text data and voice data, and classifying and evaluating those emotions.

[0887] "Means for generating processed voice that reflects emotions" refers to technologies and algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions, and generating voice data that reflects the user's emotions.

[0888] A "service representative" is a person whose role is to collect user opinions and feedback and to improve or respond to the service based on that information.

[0889] "Anonymity" refers to a state in which a user's personal information or specific characteristics are protected from being known to third parties.

[0890] MODE FOR CARRYING OUT THE INVENTION

[0891] The present invention relates to a system for generating processed speech that reflects emotions by removing characteristics of a user's speech. Specific embodiments of this system will be described below.

[0892] System Overview

[0893] The system receives the user's voice data and converts it into text using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data, analyzes the user's emotions using an emotion engine, and finally uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[0894] Hardware and software used

[0895] Speech recognition software: Google Cloud Speech-to-Text API

[0896] Natural language processing libraries: spaCy, NLTK

[0897] Emotion engine: IBM Watson Tone Analyzer

[0898] Text-to-speech software: Amazon Polly

[0899] Specific examples

[0900] Consider the case where a user says, "Well, then, I don't think the quality of product A is good." The system works as follows.

[0901] 1. User voice input: The user speaks into the smartphone microphone, "Well, then, I don't think the quality of product A is good."

[0902] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[0903] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the speech data into text data such as, "Well, then, I don't think the quality of product A is good."

[0904] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Well, then," from the text data, generating text data such as "I don't think the quality of product A is good."

[0905] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[0906] 6. Generate processed speech: The server uses Amazon Polly to convert the text data, "I think the quality of product A is poor," into processed speech with a tone that reflects sadness.

[0907] 7. Sending the processed voice: The server sends the generated processed voice to the service representative, who can anonymously listen to the user's opinion through this processed voice.

[0908] Prompt Sentence Examples

[0909] An example of a prompt to input to a generative AI model is as follows:

[0910] User says: "Well, then, I don't think product A is of good quality."

[0911] Prompt for generative AI model: "Remove the user's utterances and generate a modified voice that reflects their emotions."

[0912] In this way, service personnel can listen to users' opinions anonymously and prevent their anonymity from being compromised by their characteristic habits or phrases. Furthermore, by processing the voice to reflect their emotions, service personnel can accurately understand the user's feelings.

[0913] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0914] Step 1:

[0915] The user inputs speech.

[0916] A user inputs voice using a device (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the device in a digital format (e.g., WAV or MP3).

[0917] Step 2:

[0918] The server receives the audio data.

[0919] The server receives the voice data sent from the device. The received voice data is stored in the server's storage. The input is the voice data from the device, and the output is the voice data stored in the server.

[0920] Step 3:

[0921] The server converts the voice data into text data.

[0922] The server uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[0923] Step 4:

[0924] The server removes characteristic habits and phrases from the text data.

[0925] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data with the characteristic habits and phrases removed. Specifically, unnecessary phrases such as "Well, then," are removed, and the text data that reads "I don't think the quality of product A is good" is generated.

[0926] Step 5:

[0927] The server analyzes the sentiment of the text data.

[0928] The server uses an emotion engine (for example, IBM Watson Tone Analyzer) to analyze the user's emotions from the text data. The input is text data from which characteristic habits and phrases have been removed, and the output is analyzed emotion data. Specifically, emotions such as "sad" are analyzed from the text data.

[0929] Step 6:

[0930] The server generates processed voice that reflects the emotion.

[0931] The server uses speech synthesis software (e.g., Amazon Polly) to generate processed speech that reflects emotions. The input is the analyzed emotion data and text data, and the output is processed speech that reflects emotions. Specifically, the text data, "I think the quality of product A is not good," is converted into processed speech with a tone that reflects sadness.

[0932] Step 7:

[0933] The server sends the processed voice to the service representative.

[0934] The server sends the generated processed voice to the service representative. The input is the processed voice that reflects the emotion, and the output is the processed voice sent to the service representative. The service representative can listen to the user's opinion anonymously through this processed voice.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0937] Conventional anonymous reporting systems have had problems such as the possibility that a user's distinctive habits or phrases could compromise anonymity, and it is difficult to accurately convey the user's feelings. This can lead to the credibility and urgency of the report not being properly assessed.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0939] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for detecting the user's emotions and adjusting the processed voice based on the emotions, and means for transmitting the adjusted processed voice, thereby enabling anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[0940] "Characteristics of a user's conversation" refers to characteristics and habits of the user's voice when speaking, specific phrases, and the like.

[0941] "Removing noise" refers to a process of removing the characteristics of a user's speech from the voice data.

[0942] "Processed voice" refers to voice data generated after removing features of the user's speech.

[0943] "Means for transmitting" refers to a function for transmitting the generated processed voice to another device or server.

[0944] "Detecting emotions" refers to the process of analyzing emotions from the content of a user's speech and the tone of their voice.

[0945] "Adjusting the processed voice based on emotion" refers to processing voice data to reflect the detected emotion.

[0946] "Making an anonymous report" refers to the act of sending a report while concealing the user's personal information and characteristics.

[0947] A system for implementing this invention detects characteristics of a user's speech, deletes the characteristics as noise, generates processed speech that does not include the deleted characteristics, detects the user's emotions, adjusts the processed speech based on the emotions, and finally transmits the adjusted processed speech.

[0948] Hardware and software used

[0949] Hardware: Smartphone

[0950] Software: Python, speech_recognition library, pydub library, librosa library, soundfile library

[0951] Processing flow

[0952] 1. Speech recognition: The user's voice is recorded using the smartphone's microphone, and text is extracted from the voice data using the speech_recognition library. This allows the user's speech to be obtained as text data.

[0953] 2. Voice processing: Using the librosa library, we remove the user's characteristic habits and phrases from the voice data. Specifically, we analyze the characteristics of the voice and remove the characteristic parts as noise.

[0954] 3. Emotion detection: Using the librosa library, the tone and pitch of the voice data are analyzed to detect the user's emotion. For example, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0955] 4. Emotion-based audio adjustment: Reprocessing audio data based on detected emotions, for example slowing down the audio to emphasize sadness or raising the pitch to emphasize surprise.

[0956] 5. Audio transmission: The processed audio data is saved using the soundfile library and sent to the reporting center using the smartphone's communication function.

[0957] Specific examples

[0958] If a user reports, "I saw a suspicious person in my neighborhood," the system generates a processed voice saying, "I saw a suspicious person," and sends it to the reporting center. Furthermore, if the user is feeling anxious, the system can emphasize that feeling.

[0959] Prompt Sentence Examples

[0960] When a user reports "I saw a suspicious person in my neighborhood," the application generates a processed voice saying "I saw a suspicious person" and sends it to the reporting center. Furthermore, if the user is feeling anxious, the application can emphasize the emotion of anxiety.

[0961] In this way, anonymous reporting that accurately conveys emotions is possible while maintaining the anonymity of the user.

[0962] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0963] Step 1:

[0964] The user records audio using the smartphone microphone. The input is the user's speech, and the output is an audio file (e.g., input.wav). This audio file is used in subsequent processing steps.

[0965] Step 2:

[0966] The device uses the speech_recognition library to extract text from an audio file. The input is an audio file, and the output is text data (e.g., "I saw a suspicious person in my neighborhood"). This text data is used to capture the user's spoken content as text information.

[0967] Step 3:

[0968] The device uses the librosa library to remove the user's characteristic habits and phrases from the voice data. The input is an audio file, and the output is the voice data with the characteristics removed. Specifically, the device analyzes the characteristics of the voice and removes the characteristic parts as noise.

[0969] Step 4:

[0970] The device uses the librosa library to analyze the tone and pitch of the voice data to detect the user's emotions. The input is an audio file, and the output is the detected emotion (e.g., "anxiety"). Specifically, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0971] Step 5:

[0972] The device reprocesses the audio data based on the detected emotion. The input is the feature-removed audio data and the detected emotion, and the output is processed audio with the emotion emphasized. For example, the audio may be slowed down to emphasize sadness, or the pitch may be raised to emphasize surprise.

[0973] Step 6:

[0974] The device uses the soundfile library to save the processed audio data. The input is the processed audio with enhanced emotion, and the output is the saved audio file (e.g., output.wav). This audio file is used to send to the reporting center.

[0975] Step 7:

[0976] The device uses the smartphone's communication function to send the processed audio file to the reporting center. The input is the saved audio file, and the output is the audio data sent to the reporting center. This allows anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[0977] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0978] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0979] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[0980] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0981] [Third embodiment]

[0982] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0983] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0984] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0985] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0986] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0988] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0989] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0990] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the headset type terminal 314, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48.

[0993] The reception and output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception and output program 60 executed on the RAM 48.

[0994] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0995] "Example 1"

[0996] One aspect of the present invention is a system that uses a generation AI to analyze a user's conversation in real time. This system detects characteristics of the user's conversation, such as phrases like "um" and "well then," and removes them as noise. The removal occurs the moment the generation AI detects a characteristic phrase, resulting in the generation of processed speech that does not contain the characteristic phrase. The generated processed speech is then transmitted to the other party through the system.

[0997] "Example 2"

[0998] As a specific example, a service representative may want to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the user's speech characteristics. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask for the user's opinion without losing their anonymity due to the user's distinctive mannerisms or phrases.

[0999] The processing flow of each embodiment will be described below.

[1000] "Example 1"

[1001] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[1002] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[1003] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[1004] Step 4: A processed voice without any features is generated and sent to the other party through the system.

[1005] "Example 2"

[1006] Step 1: When a service representative wants to anonymously ask a user for their opinion, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[1007] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[1008] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[1009] Step 4: A processed voice without the user's characteristics is generated and sent to the service representative. This allows the service representative to listen to the user's opinions anonymously and prevents the user's characteristic habits and phrases from compromising the anonymity.

[1010] Example 1

[1011] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] Conventional voice communication systems have a problem in that characteristic phrases such as "umm" and "well then" contained in user conversations remain as noise, making the voice difficult for the other party to hear. In addition, manually removing this noise requires a great deal of effort, making it impractical for real-time conversations. Furthermore, even when collecting user opinions anonymously, the presence of noise makes it difficult to accurately grasp their opinions.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1014] In this invention, the server includes means for detecting features of a user's speech, means for removing the features as noise, and means for generating processed speech that does not include the removed features, thereby making it possible to remove noise from a user's speech in real time and generate clear speech.

[1015] "User" refers to an individual or entity that uses the system to engage in conversations.

[1016] "Conversational features" refer to specific phrases or words such as "um" or "well then" that are included in the user's speech.

[1017] "Noise" refers to unnecessary phrases and words that are detected as characteristics of conversation and are targeted for removal.

[1018] "Processed audio" refers to clear audio data after noise has been removed.

[1019] "Server" refers to a computer system for processing and analyzing voice data.

[1020] "Terminal" refers to a device used by a user that records and plays audio.

[1021] "Speech recognition software" refers to a program for converting voice data into text data.

[1022] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[1023] "Text data" refers to textual information converted by voice recognition software.

[1024] "Clean text data" refers to text data after noise has been removed.

[1025] "Voice data" refers to data that is a digital representation of a user's speech.

[1026] "Real-time" refers to processing occurring simultaneously with the user's conversation.

[1027] The present invention relates to a system for analyzing user conversations in real time and removing noise. Specific embodiments of this system will be described below.

[1028] System configuration

[1029] Hardware

[1030] Server: Uses a computer system equipped with high-performance CPUs and GPUs, which allows for large amounts of data processing and the execution of generative AI models.

[1031] Device: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[1032] software

[1033] Speech Recognition Software: We use common speech recognition APIs (e.g., Google Speech-to-Text API) to convert voice data into text data.

[1034] Generative AI models: Use generative AI models (e.g., OpenAI's GPT-3) to analyze user conversations and detect characteristic phrases.

[1035] System Operation

[1036] Acquiring and sending audio data

[1037] When a user starts talking, the device uses a built-in microphone to record the user's voice, and the recorded voice data is sent to the server in real time.

[1038] Converting audio data to text

[1039] The server uses speech recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted directly into text data.

[1040] Text data analysis and noise removal

[1041] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. These phrases are removed as noise, and clean text data is generated.

[1042] Speech conversion of clean text data

[1043] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[1044] Sending and playing processed audio

[1045] The generated processed voice is sent from the server to the terminal, which then plays it back, allowing the user to hear clearer voice with the noise removed.

[1046] Examples of specific examples and prompts

[1047] For example, suppose a user has the following conversation:

[1048] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1049] The server receives this conversation and converts it into the following text data using speech recognition software:

[1050] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1051] Next, we use a generative AI model to remove noise as follows:

[1052] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[1053] Finally, the server converts the cleaned text data back into audio data, generates processed audio, and sends it to the terminal, which plays the processed audio.

[1054] An example of a prompt sentence might be:

[1055] "Detect distinctive phrases in user conversations and remove them as noise, such as 'um' or 'Well then.'"

[1056] By inputting this prompt into a generative AI model, a process is performed to remove noise from the user's conversation.

[1057] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1058] Step 1:

[1059] A user initiates a conversation.

[1060] The user begins speaking to the system, for example, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[1061] Step 2:

[1062] The device records the user's voice and sends it to the server.

[1063] The device records the user's voice using a built-in microphone. The recorded voice data is sent to the server in real time. The input is the user's voice data, and the output is the voice data sent to the server.

[1064] Step 3:

[1065] The server uses voice recognition software to convert the voice data into text data.

[1066] The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[1067] Step 4:

[1068] The server uses the generative AI model to analyze the text data and detect characteristic phrases.

[1069] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[1070] Step 5:

[1071] The server removes the detected phrases as noise and generates clean text data.

[1072] The server removes the detected characteristic phrases as noise. For example, from the text data "Um, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Um" and "So, let's go for a walk in the afternoon" to generate clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[1073] Step 6:

[1074] The server reconverts the clean text data into audio data.

[1075] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk this afternoon" is generated. The input is cleaned text data, and the output is speech data.

[1076] Step 7:

[1077] The server transmits the generated processed voice to the terminal.

[1078] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[1079] Step 8:

[1080] The device plays the processed audio.

[1081] The device then plays back the processed audio, allowing the user to hear clear audio with the noise removed. The input is the audio data sent to the device, and the output is the audio played back from the device's speaker.

[1082] (Application example 1)

[1083] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1084] Conventional audio distribution systems have the problem that noises such as "umm" and "well then" that occur during a streamer's conversation are transmitted to the listener, degrading the quality of the audio. Streamers are also required to consciously speak to avoid the noise, making it difficult to have a natural conversation.

[1085] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed audio that does not include the removed features. This makes it possible to remove noise from conversation in real time during live streaming or audio distribution. Furthermore, by including means for generating new text based on text from which noise has been removed using a generative AI model, the broadcaster can provide clear audio to listeners while continuing a natural conversation without having to consciously avoid noise.

[1086] "User speech characteristics" refers to specific phrases and word patterns that a user uses when speaking.

[1087] "Noise" refers to phrases or words that are unnecessary or unwanted in a conversation.

[1088] "Removal" refers to the process of removing detected noise.

[1089] "Processed audio" refers to the clear audio after noise has been removed.

[1090] "Sending" refers to the process of delivering the generated processed voice to the other party.

[1091] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[1092] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[1093] "Generative AI model" refers to an algorithm or system that uses artificial intelligence to generate text or speech.

[1094] "New text" refers to the text generated by the generative AI model after noise has been removed.

[1095] "Listener" refers to a person who receives and listens to the distributed audio.

[1096] The system for carrying out this invention detects the characteristics of a user's speech, removes them as noise, generates processed speech, and transmits it. A specific embodiment of this system will be described below.

[1097] System configuration

[1098] This system uses the following hardware and software:

[1099] Hardware: Microphone, Smartphone

[1100] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (Hugging Face GPT-3 model)

[1101] Processing flow

[1102] 1. Speech recognition: The user's speech through the microphone is converted into text using the SpeechRecognition library, which allows the user's speech characteristics to be obtained as text data.

[1103] 2. Noise removal: Remove noise phrases such as "um" and "well then" from the acquired text data. This process is done by detecting specific phrases and removing them from the text.

[1104] 3. Generative AI model: Based on the noise-removed text data, a generative AI model (GPT-3) is used to generate new text. This new text will have clear content with noise removed.

[1105] 4. Generate processed speech: The new text is converted into speech and generated as processed speech, which is a clear speech with noise removed.

[1106] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive clear audio.

[1107] Specific examples

[1108] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide the stream to listeners as, "Today I'm going to talk about my new project."

[1109] Prompt Sentence Examples

[1110] Below are some example prompts to input to the generative AI model:

[1111] Remove noises like "um" and "well then" from user conversations to produce clearer voices.

[1112] By inputting this prompt into a generative AI model, clear audio with noise removed can be generated.

[1113] In this way, the present invention removes conversation noise in real time during live streaming or audio distribution, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[1114] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1115] Step 1:

[1116] The device collects the user's voice as they speak through a microphone. The device then converts the collected voice data into text data using the SpeechRecognition library. This process converts the voice data into text data. The input is voice data, and the output is text data.

[1117] Step 2:

[1118] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes any matching phrases. This process results in clean text data with the noise removed. The input is text data containing noise, and the output is text data with the noise removed.

[1119] Step 3:

[1120] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, it uses prompt sentences to give instructions to the generative AI model, which then generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new text data that has been generated.

[1121] Step 4:

[1122] The device inputs the generated new text data into a speech synthesis engine to generate processed speech. Specifically, the text data is passed to the speech synthesis engine, which outputs it as speech data. This process results in clear speech data with noise removed. The input is the generated new text data, and the output is processed speech data.

[1123] Step 5:

[1124] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[1125] Example 2

[1126] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1127] In conventional systems, the user's characteristic conversational habits and phrases remain intact, which can compromise anonymity. Furthermore, when collecting user opinions as text, manual conversion work is required, which is inefficient. This makes it difficult to completely protect the user's privacy when service personnel anonymously ask employees for user opinions.

[1128] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, means for generating processed voice that does not include the removed characteristics, means for converting the processed voice into text, and means for transmitting the text. This makes it possible to remove characteristic habits and phrases of the user and efficiently collect user opinions as text while maintaining anonymity.

[1129] "User conversation characteristics" refers to audio characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[1130] "Noise" refers to unnecessary features of a user's conversation that may compromise anonymity.

[1131] "Processed voice" refers to voice data generated by removing noise characteristic of a user's conversation and using the remaining content.

[1132] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[1133] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[1134] "Means for converting to text" refers to the technology or system for analyzing audio data and converting its content into text information.

[1135] "Transmitting means" refers to the technology or system used to transmit the generated text or audio data to other devices or systems.

[1136] A "service representative" is a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[1137] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[1138] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[1139] MODE FOR CARRYING OUT THE INVENTION

[1140] The present invention provides a system for detecting characteristics of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[1141] Hardware and software used

[1142] Hardware: Servers, devices (PCs, smartphones, etc.)

[1143] Software: Generative AI models (e.g., GPT-4), audio processing software (e.g., Audacity, Adobe Audition)

[1144] System Overview

[1145] The user uses a device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server then passes the received audio file through voice processing software, which removes the user's distinctive mannerisms and phrases. The removed audio is then generated as processed voice.

[1146] Next, the server inputs a prompt sentence to the generative AI model to convert the processed speech into text. For example, the prompt sentence can be "Please convert the following speech into text: [processed speech]." The generative AI model converts the speech to text based on the prompt sentence and returns the result to the server. The server then sends the converted text to the service representative. The service representative reviews the received text and anonymously understands the user's opinion.

[1147] Specific examples

[1148] Consider a scenario where a user says, "Well, then, I don't think product A is of good quality." When the user speaks into the device's microphone, the device records this speech and saves it as a digital audio file in WAV or MP3 format. The file name will include the date and time, such as "user_input_20231010.wav."

[1149] The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload." The server passes the received audio file to its voice processing software, which removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of Product A is good."

[1150] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates text saying, "I think the quality of product A is not good," and returns it to the server. The server then sends this text to the service representative via email or chat app. The transmitted content is, "User's opinion: I think the quality of product A is not good."

[1151] The service representative reviews the received text and anonymously captures the user's opinion, which removes the user's characteristic habits and phrases.

[1152] This system allows service personnel to collect user opinions anonymously and prevents users' characteristic habits and phrases from compromising that anonymity.

[1153] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1154] Step 1:

[1155] The user provides voice input. The user speaks into the device's microphone. For example, the user might say, "Well, then, I think the quality of product A is not good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[1156] Step 2:

[1157] The device records the audio and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's audio data, and the output is the saved digital audio file.

[1158] Step 3:

[1159] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The input is a digital audio file, and the output is the audio file sent to the server.

[1160] Step 4:

[1161] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server then passes the received audio file to audio processing software (e.g., Audacity's script function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[1162] Step 5:

[1163] The server inputs a prompt to the generative AI model to convert the processed speech into text. The server inputs the prompt to the generative AI model (e.g., GPT-4) as "Please convert the following speech into text: [processed speech]". The processed speech is attached to the prompt as binary data. The input is the processed speech and the prompt, and the output is the prompt sent to the generative AI model.

[1164] Step 6:

[1165] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and processed speech, and the output is the generated text.

[1166] Step 7:

[1167] The server sends the converted text to the service representative. The server then sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[1168] Step 8:

[1169] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This allows for opinions that have been removed of the user's characteristic habits and phrases. The input is the text sent to the service representative, and the output is the confirmed user opinion.

[1170] (Application example 2)

[1171] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1172] Conventional opinion collection systems leave users' characteristic habits and phrases intact, which can compromise anonymity. It is also difficult to collect opinions in real time, making it difficult to efficiently gather feedback. This makes it difficult for users to provide honest opinions, and there is the issue of how difficult it is to obtain the information needed to improve the service.

[1173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1174] In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed speech that does not include the removed features, thereby enabling efficient collection of opinions in real time while maintaining user anonymity.

[1175] "User" means any person or entity that uses the System to provide input.

[1176] "Conversational features" are individually identifiable elements such as habits, phrases, and intonation contained in a user's speech.

[1177] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[1178] "Removal" refers to the process of removing detected noise.

[1179] "Processed audio" is audio data from which noise has been removed.

[1180] "Send" is the act of transferring the processed audio to a designated recipient.

[1181] A "system" is a set of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[1182] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can also connect to the Internet and run applications.

[1183] An "application" is a software program that runs on a smartphone.

[1184] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[1185] "Real-time" refers to data processing and analysis occurring immediately.

[1186] "Security services" are services aimed at protecting information and ensuring anonymity.

[1187] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[1188] The system for implementing this invention executes a series of processes: detecting the characteristics of a user's speech, removing them as noise, generating processed speech, and transmitting it. Specifically, this is realized using an application installed on a smartphone.

[1189] Hardware and software used

[1190] Hardware: Smartphone

[1191] Software: Python, speech_recognition library, gTTS library

[1192] System Operation

[1193] 1. Speech Recognition: An application installed on a smartphone collects the user's speech, which is then converted into text using the speech_recognition library. This process uses a generative AI model to analyze the user's speech in real time.

[1194] 2. Text processing: The converted text is then stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "Well then," etc.).

[1195] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, which is a modified voice that removes the user's characteristic mannerisms and phrases.

[1196] 4. Transmission: The generated processed voice is transmitted to security service personnel, allowing them to collect opinions while maintaining the user's anonymity.

[1197] Specific examples

[1198] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[1199] Prompt Sentence Examples

[1200] Example prompts to input to a generative AI model:

[1201] Convert your users' speech to text and remove quirks and phrases to create an anonymous audio recording.

[1202] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[1203] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1204] Step 1:

[1205] The user launches the application installed on their smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[1206] Step 2:

[1207] The device converts collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze user speech in real time. Input: Collected voice data. Output: Converted text data.

[1208] Step 3:

[1209] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[1210] Step 4:

[1211] The device uses the gTTS library to convert the noise-removed text data back into speech. This speech is a processed speech that removes the user's characteristic habits and phrases. Input: Text data with noise removed. Output: Processed speech data.

[1212] Step 5:

[1213] The device sends the generated processed voice data to the security service staff. This allows the staff to collect opinions while maintaining the anonymity of the user. Input: Processed voice data. Output: Processed voice data sent to the staff.

[1214] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1215] "Example 1"

[1216] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system detects features of a user's speech, removes the features as noise, generates processed speech that does not include the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the removal of features based on the emotions. For example, if a user is angry,

[1217] If the user is perceived as angry, the emotion engine may emphasize the angry features. If the user is perceived as happy, the emotion engine may emphasize the happy features. This will more accurately convey the user's emotions.

[1218] "Example 2"

[1219] The emotion engine also adjusts the generation of processed voices based on the user's emotion. For example, if the user feels sad, the emotion engine may emphasize the sadness characteristics. If the user feels surprised, the emotion engine may emphasize the surprise characteristics. This allows the user's emotion to be conveyed more accurately.

[1220] The processing flow of each embodiment will be described below.

[1221] "Example 1"

[1222] Step 1: Input the user's conversation into the emotion engine.

[1223] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1224] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[1225] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[1226] Step 5: Send the generated processed audio.

[1227] "Example 2"

[1228] Step 1: Input the user's conversation into the emotion engine.

[1229] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1230] Step 3: Adjust the generated processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[1231] Step 4: Generate processed audio based on the emotion-based adjustment results.

[1232] Step 5: Send the generated processed audio.

[1233] Example 1

[1234] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1235] Conventional voice analysis systems have had problems in effectively removing noises such as "umm" and "well then" that are included in user conversations, and also in not being able to properly reflect the user's emotions. As a result, the user's intentions and emotions are often not conveyed accurately.

[1236] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, means for generating processed voice that does not include the removed features, means for transmitting the processed voice, and means for analyzing the user's emotions and adjusting the removal of features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively removing noise from the user's conversation.

[1237] "User" refers to an individual or entity that uses the system to engage in conversations.

[1238] "Conversational features" refer to specific phrases and word patterns contained in a user's speech.

[1239] "Noise" refers to unnecessary phrases and words that disrupt the flow of conversation.

[1240] "Removal" refers to the process of removing detected noise from audio data.

[1241] "Processed audio" refers to audio data after noise has been removed.

[1242] "Transmit" refers to transferring the generated processed audio to another device or system.

[1243] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[1244] "Adjustment" refers to changing the noise removal method based on the results of sentiment analysis.

[1245] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases and words.

[1246] The present invention is a system that analyzes a user's conversation in real time, removes noise, and generates processed voice that reflects emotions. Specific embodiments of this system will be described below.

[1247] Hardware and software used

[1248] Hardware: Microphone, speaker, device (smartphone, tablet, PC)

[1249] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[1250] System configuration

[1251] Subject: Terminal

[1252] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[1253] Subject: Server

[1254] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects and removes noises such as "um" and "well then" from the text data. After the deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of removing the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, the "um" may not be removed to emphasize the anger.

[1255] The server converts the noise-removed text data back into audio data using text-to-speech synthesis technology, and the processed audio is sent from the server to the device.

[1256] Subject: Terminal

[1257] The device then plays the processed voice to the user through a speaker, allowing the user to hear the voice with noise removed and emotion reflected.

[1258] Specific examples

[1259] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. The emotion engine analyzes the user's emotions, and if it determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying, "Hello, it's a nice day today," is generated and sent to the device. The device then plays this processed voice back to the user.

[1260] Prompt Sentence Examples

[1261] "Detect characteristic phrases such as 'um' and 'well then' from the user's conversation and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, emphasize the angry features, and if the user is happy, emphasize the happy features."

[1262] The above is a specific embodiment for carrying out the present invention.

[1263] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1264] Step 1: Collecting audio data

[1265] Subject: Terminal

[1266] Input: User conversation

[1267] Specific operation: When the user starts speaking, the microphone on the device will automatically collect the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data will be recorded on the device.

[1268] Output: Collected audio data

[1269] Step 2: Sending audio data

[1270] Subject: Terminal

[1271] Input: Collected audio data

[1272] How it works: The device sends the collected voice data to the server in real time using an internet connection, such as Wi-Fi or mobile data.

[1273] Output: Audio data sent to the server

[1274] Step 3: Analyzing the audio data

[1275] Subject: Server

[1276] Input: Audio data sent to the server

[1277] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the generated text is "Hello, um, it's a nice day today."

[1278] Output: Text data

[1279] Step 4: Detect and remove noise

[1280] Subject: Server

[1281] Input: Text data

[1282] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" or "well then" from the text data. For example, the generated text would be "Hello, it's a nice day today."

[1283] Output: Text data with noise removed

[1284] Step 5: Sentiment analysis and adjustment

[1285] Subject: Server

[1286] Input: Text data with noise removed

[1287] Specific operation: The server analyzes the user's emotions using the emotion engine. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the server may not remove "um" to emphasize the angry characteristic.

[1288] Output: Text data reflecting emotions

[1289] Step 6: Generate processed audio

[1290] Subject: Server

[1291] Input: Text data that reflects emotions

[1292] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data such as "Hello, it's a nice day today" is generated.

[1293] Output: Processed audio data

[1294] Step 7: Send your processed audio

[1295] Subject: Server

[1296] Input: processed audio data

[1297] Specific operation: The server sends the generated processed audio to the device. This is done via an internet connection, such as Wi-Fi or mobile data.

[1298] Output: Processed audio data sent to the device

[1299] Step 8: Play the processed audio

[1300] Subject: Terminal

[1301] Input: Processed audio data sent to the device

[1302] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear "Hello, it's a nice day today."

[1303] Output: The processed audio played to the user

[1304] (Application example 1)

[1305] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1306] Conventional audio content distribution systems have the problem that unnecessary phrases and noise contained in the broadcaster's conversation are distributed as is, making the content difficult for listeners to hear. Also, since the broadcaster's emotions are not accurately conveyed, it is difficult to provide emotionally rich content to listeners.

[1307] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1308] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed audio that does not include the deleted features, means for transmitting the processed audio, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed audio, and means for transmitting the generated processed audio. This makes it possible to delete unnecessary phrases from the distributor's conversation and provide listeners with audio content that emphasizes emotions in real time.

[1309] "User conversation characteristics" refers to specific phrases and word patterns included when a user speaks.

[1310] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[1311] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[1312] "Generative AI" refers to a system that uses artificial intelligence technology to generate text or speech.

[1313] An "emotion engine" is a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[1314] "Audio Content Provider" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[1315] A "listener" refers to a user who receives, watches, or listens to audio content.

[1316] "Real-time" refers to processing occurring simultaneously as the user speaks.

[1317] "Adjusting feature removal" refers to dynamically changing the selection of phrases and words to be removed based on the user's sentiment.

[1318] "Adjusted processed audio" refers to audio data after emotions have been emphasized and noise removed by the emotion engine.

[1319] A system for implementing this invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed speech that emphasizes emotions. A specific embodiment of this system will be described below.

[1320] System Configuration

[1321] The system consists of the following main components:

[1322] 1. Speech Recognition Module: Converts the user's speech into text.

[1323] 2. Generative AI model: Analyzes text and removes unnecessary phrases.

[1324] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[1325] 4. Speech generation module: converts the adjusted text into speech.

[1326] 5. Communication module: Transmits the generated processed audio to the listener.

[1327] Hardware and software used

[1328] Speech Recognition Module: Uses the speech_recognition library.

[1329] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[1330] Sentiment analysis engine: Uses the transformers library pipeline to utilize sentiment analysis models.

[1331] Speech generation module: Uses a TTS (Text-to-Speech) engine to convert text into speech.

[1332] Communications module: uses a communications protocol to transmit audio data over the Internet.

[1333] Data processing and calculation

[1334] 1. Speech recognition: Obtain the user's voice data and convert it to text using the speech_recognition library.

[1335] 2. Noise removal: Using a generative AI model to remove unnecessary phrases from the text.

[1336] 3. Sentiment analysis: Using a sentiment analysis engine, we analyze user sentiment from text.

[1337] 4. Sentiment adjustment: Adjust the text based on the analyzed sentiment.

[1338] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[1339] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[1340] Specific examples

[1341] For example, if a streamer says, "Well, today I'm going to talk about a new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about a new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[1342] Prompt Sentence Examples

[1343] User Spoken Text: "Well, today I'm going to talk about a new project. So, let me start by giving you an overview."

[1344] Sentiment analysis result: "Joy"

[1345] Generated text: "Today we'll be talking about a new project. Let's start with an overview."

[1346] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[1347] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1348] Step 1:

[1349] A user speaks into a microphone to deliver audio content. The device captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[1350] Step 2:

[1351] The device sends the acquired voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data into text. The input is the voice data, and the output is the converted text data.

[1352] Step 3:

[1353] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[1354] Step 4:

[1355] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[1356] Step 5:

[1357] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, it processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with emphasized emotions.

[1358] Step 6:

[1359] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[1360] Step 7:

[1361] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data via the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[1362] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and emotionally-emphasized voice content can be provided to the listener.

[1363] Example 2

[1364] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1365] Conventional systems have difficulty not only removing the characteristics of a user's speech but also generating speech that accurately reflects the user's emotions. Furthermore, when collecting user opinions anonymously, there is a risk that the user's distinctive habits and phrases may remain, compromising anonymity. Furthermore, it is difficult for service personnel to accurately understand the user's emotions.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1367] In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, and means for generating processed speech that does not include the removed characteristics. This makes it possible to remove the user's characteristic habits and phrases and collect user opinions while maintaining anonymity. The server also includes means for analyzing the user's emotions and means for generating processed speech that reflects the analyzed emotions. This allows the generated speech to accurately reflect the user's emotions, enabling the service representative to accurately understand the user's emotions.

[1368] "User conversation characteristics" refers to speech characteristics such as habits, phrases, and intonation that are specific to the user's speech.

[1369] "Noise" refers to distinctive habits and phrases in a user's conversations that may compromise anonymity.

[1370] "Processed voice" refers to voice data in which the characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[1371] "Means for analyzing emotions" refers to technologies and algorithms for detecting emotions from users' text data and voice data, and classifying and evaluating those emotions.

[1372] "Means for generating processed voice that reflects emotions" refers to technologies and algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions, and generating voice data that reflects the user's emotions.

[1373] A "service representative" is a person whose role is to collect user opinions and feedback and to improve or respond to the service based on that information.

[1374] "Anonymity" refers to a state in which a user's personal information or specific characteristics are protected from being known to third parties.

[1375] MODE FOR CARRYING OUT THE INVENTION

[1376] The present invention relates to a system for generating processed speech that reflects emotions by removing characteristics of a user's speech. Specific embodiments of this system will be described below.

[1377] System Overview

[1378] The system receives the user's voice data and converts it into text using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data, analyzes the user's emotions using an emotion engine, and finally uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[1379] Hardware and software used

[1380] Speech recognition software: Google Cloud Speech-to-Text API

[1381] Natural language processing libraries: spaCy, NLTK

[1382] Emotion engine: IBM Watson Tone Analyzer

[1383] Text-to-speech software: Amazon Polly

[1384] Specific examples

[1385] Consider the case where a user says, "Well, then, I don't think the quality of product A is good." The system works as follows.

[1386] 1. User voice input: The user speaks into the smartphone microphone, "Well, then, I don't think the quality of product A is good."

[1387] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[1388] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the speech data into text data such as, "Well, then, I don't think the quality of product A is good."

[1389] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Well, then," from the text data, generating text data such as "I don't think the quality of product A is good."

[1390] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[1391] 6. Generate processed speech: The server uses Amazon Polly to convert the text data, "I think the quality of product A is poor," into processed speech with a tone that reflects sadness.

[1392] 7. Sending the processed voice: The server sends the generated processed voice to the service representative, who can anonymously listen to the user's opinion through this processed voice.

[1393] Prompt Sentence Examples

[1394] An example of a prompt to input to a generative AI model is as follows:

[1395] User says: "Well, then, I don't think product A is of good quality."

[1396] Prompt for generative AI model: "Remove the user's utterances and generate a modified voice that reflects their emotions."

[1397] In this way, service personnel can listen to users' opinions anonymously and prevent their anonymity from being compromised by their characteristic habits or phrases. Furthermore, by processing the voice to reflect their emotions, service personnel can accurately understand the user's feelings.

[1398] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1399] Step 1:

[1400] The user inputs speech.

[1401] A user inputs voice using a device (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the device in a digital format (e.g., WAV or MP3).

[1402] Step 2:

[1403] The server receives the audio data.

[1404] The server receives the voice data sent from the device. The received voice data is stored in the server's storage. The input is the voice data from the device, and the output is the voice data stored in the server.

[1405] Step 3:

[1406] The server converts the voice data into text data.

[1407] The server uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[1408] Step 4:

[1409] The server removes characteristic habits and phrases from the text data.

[1410] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data with the characteristic habits and phrases removed. Specifically, unnecessary phrases such as "Well, then," are removed, and the text data that reads "I don't think the quality of product A is good" is generated.

[1411] Step 5:

[1412] The server analyzes the sentiment of the text data.

[1413] The server uses an emotion engine (for example, IBM Watson Tone Analyzer) to analyze the user's emotions from the text data. The input is text data from which characteristic habits and phrases have been removed, and the output is analyzed emotion data. Specifically, emotions such as "sad" are analyzed from the text data.

[1414] Step 6:

[1415] The server generates processed voice that reflects the emotion.

[1416] The server uses speech synthesis software (e.g., Amazon Polly) to generate processed speech that reflects emotions. The input is the analyzed emotion data and text data, and the output is processed speech that reflects emotions. Specifically, the text data, "I think the quality of product A is not good," is converted into processed speech with a tone that reflects sadness.

[1417] Step 7:

[1418] The server sends the processed voice to the service representative.

[1419] The server sends the generated processed voice to the service representative. The input is the processed voice that reflects the emotion, and the output is the processed voice sent to the service representative. The service representative can listen to the user's opinion anonymously through this processed voice.

[1420] (Application example 2)

[1421] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1422] Conventional anonymous reporting systems have had problems such as the possibility that a user's distinctive habits or phrases could compromise anonymity, and it is difficult to accurately convey the user's feelings. This can lead to the credibility and urgency of the report not being properly assessed.

[1423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1424] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for detecting the user's emotions and adjusting the processed voice based on the emotions, and means for transmitting the adjusted processed voice, thereby enabling anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[1425] "Characteristics of a user's conversation" refers to characteristics and habits of the user's voice when speaking, specific phrases, and the like.

[1426] "Removing noise" refers to a process of removing the characteristics of a user's speech from the voice data.

[1427] "Processed voice" refers to voice data generated after removing features of the user's speech.

[1428] "Means for transmitting" refers to a function for transmitting the generated processed voice to another device or server.

[1429] "Detecting emotions" refers to the process of analyzing emotions from the content of a user's speech and the tone of their voice.

[1430] "Adjusting the processed voice based on emotion" refers to processing voice data to reflect the detected emotion.

[1431] "Making an anonymous report" refers to the act of sending a report while concealing the user's personal information and characteristics.

[1432] A system for implementing this invention detects characteristics of a user's speech, deletes the characteristics as noise, generates processed speech that does not include the deleted characteristics, detects the user's emotions, adjusts the processed speech based on the emotions, and finally transmits the adjusted processed speech.

[1433] Hardware and software used

[1434] Hardware: Smartphone

[1435] Software: Python, speech_recognition library, pydub library, librosa library, soundfile library

[1436] Processing flow

[1437] 1. Speech recognition: The user's voice is recorded using the smartphone's microphone, and text is extracted from the voice data using the speech_recognition library. This allows the user's speech to be obtained as text data.

[1438] 2. Voice processing: Using the librosa library, we remove the user's characteristic habits and phrases from the voice data. Specifically, we analyze the characteristics of the voice and remove the characteristic parts as noise.

[1439] 3. Emotion detection: Using the librosa library, the tone and pitch of the voice data are analyzed to detect the user's emotion. For example, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[1440] 4. Emotion-based audio adjustment: Reprocessing audio data based on detected emotions, for example slowing down the audio to emphasize sadness or raising the pitch to emphasize surprise.

[1441] 5. Audio transmission: The processed audio data is saved using the soundfile library and sent to the reporting center using the smartphone's communication function.

[1442] Specific examples

[1443] If a user reports, "I saw a suspicious person in my neighborhood," the system generates a processed voice saying, "I saw a suspicious person," and sends it to the reporting center. Furthermore, if the user is feeling anxious, the system can emphasize that feeling.

[1444] Prompt Sentence Examples

[1445] When a user reports "I saw a suspicious person in my neighborhood," the application generates a processed voice saying "I saw a suspicious person" and sends it to the reporting center. Furthermore, if the user is feeling anxious, the application can emphasize the emotion of anxiety.

[1446] In this way, anonymous reporting that accurately conveys emotions is possible while maintaining the anonymity of the user.

[1447] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1448] Step 1:

[1449] The user records audio using the smartphone microphone. The input is the user's speech, and the output is an audio file (e.g., input.wav). This audio file is used in subsequent processing steps.

[1450] Step 2:

[1451] The device uses the speech_recognition library to extract text from an audio file. The input is an audio file, and the output is text data (e.g., "I saw a suspicious person in my neighborhood"). This text data is used to capture the user's spoken content as text information.

[1452] Step 3:

[1453] The device uses the librosa library to remove the user's characteristic habits and phrases from the voice data. The input is an audio file, and the output is the voice data with the characteristics removed. Specifically, the device analyzes the characteristics of the voice and removes the characteristic parts as noise.

[1454] Step 4:

[1455] The device uses the librosa library to analyze the tone and pitch of the voice data to detect the user's emotions. The input is an audio file, and the output is the detected emotion (e.g., "anxiety"). Specifically, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[1456] Step 5:

[1457] The device reprocesses the audio data based on the detected emotion. The input is the feature-removed audio data and the detected emotion, and the output is processed audio with the emotion emphasized. For example, the audio may be slowed down to emphasize sadness, or the pitch may be raised to emphasize surprise.

[1458] Step 6:

[1459] The device uses the soundfile library to save the processed audio data. The input is the processed audio with enhanced emotion, and the output is the saved audio file (e.g., output.wav). This audio file is used to send to the reporting center.

[1460] Step 7:

[1461] The device uses the smartphone's communication function to send the processed audio file to the reporting center. The input is the saved audio file, and the output is the audio data sent to the reporting center. This allows anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[1462] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1464] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1465] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1466] [Fourth embodiment]

[1467] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1468] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1469] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1470] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1471] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1472] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1473] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1474] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1475] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1478] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1479] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1480] "Example 1"

[1481] One aspect of the present invention is a system that uses a generation AI to analyze a user's conversation in real time. This system detects characteristics of the user's conversation, such as phrases like "um" and "well then," and removes them as noise. The removal occurs the moment the generation AI detects a characteristic phrase, resulting in the generation of processed speech that does not contain the characteristic phrase. The generated processed speech is then transmitted to the other party through the system.

[1482] "Example 2"

[1483] As a specific example, a service representative may want to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the user's speech characteristics. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask for the user's opinion without losing their anonymity due to the user's distinctive mannerisms or phrases.

[1484] The processing flow of each embodiment will be described below.

[1485] "Example 1"

[1486] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[1487] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[1488] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[1489] Step 4: A processed voice without any features is generated and sent to the other party through the system.

[1490] "Example 2"

[1491] Step 1: When a service representative wants to anonymously ask a user for their opinion, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[1492] Step 2: Generative AI analyzes the user's conversation in real time, including detecting distinctive phrases and habits.

[1493] Step 3: The generative AI removes characteristic phrases and habits as noise. This removal occurs the moment characteristic phrases and habits are detected.

[1494] Step 4: A processed voice without the user's characteristics is generated and sent to the service representative. This allows the service representative to listen to the user's opinions anonymously and prevents the user's characteristic habits and phrases from compromising the anonymity.

[1495] Example 1

[1496] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1497] Conventional voice communication systems have a problem in that characteristic phrases such as "umm" and "well then" contained in user conversations remain as noise, making the voice difficult for the other party to hear. In addition, manually removing this noise requires a great deal of effort, making it impractical for real-time conversations. Furthermore, even when collecting user opinions anonymously, the presence of noise makes it difficult to accurately grasp their opinions.

[1498] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1499] In this invention, the server includes means for detecting features of a user's speech, means for removing the features as noise, and means for generating processed speech that does not include the removed features, thereby making it possible to remove noise from a user's speech in real time and generate clear speech.

[1500] "User" refers to an individual or entity that uses the system to engage in conversations.

[1501] "Conversational features" refer to specific phrases or words such as "um" or "well then" that are included in the user's speech.

[1502] "Noise" refers to unnecessary phrases and words that are detected as characteristics of conversation and are targeted for removal.

[1503] "Processed audio" refers to clear audio data after noise has been removed.

[1504] "Server" refers to a computer system for processing and analyzing voice data.

[1505] "Terminal" refers to a device used by a user that records and plays audio.

[1506] "Speech recognition software" refers to a program for converting voice data into text data.

[1507] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[1508] "Text data" refers to textual information converted by voice recognition software.

[1509] "Clean text data" refers to text data after noise has been removed.

[1510] "Voice data" refers to data that is a digital representation of a user's speech.

[1511] "Real-time" refers to processing occurring simultaneously with the user's conversation.

[1512] The present invention relates to a system for analyzing user conversations in real time and removing noise. Specific embodiments of this system will be described below.

[1513] System configuration

[1514] Hardware

[1515] Server: Uses a computer system equipped with high-performance CPUs and GPUs, which allows for large amounts of data processing and the execution of generative AI models.

[1516] Device: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[1517] software

[1518] Speech Recognition Software: We use common speech recognition APIs (e.g., Google Speech-to-Text API) to convert voice data into text data.

[1519] Generative AI models: Use generative AI models (e.g., OpenAI's GPT-3) to analyze user conversations and detect characteristic phrases.

[1520] System Operation

[1521] Acquiring and sending audio data

[1522] When a user starts talking, the device uses a built-in microphone to record the user's voice, and the recorded voice data is sent to the server in real time.

[1523] Converting audio data to text

[1524] The server uses speech recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted directly into text data.

[1525] Text data analysis and noise removal

[1526] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. These phrases are removed as noise, and clean text data is generated.

[1527] Speech conversion of clean text data

[1528] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[1529] Sending and playing processed audio

[1530] The generated processed voice is sent from the server to the terminal, which then plays it back, allowing the user to hear clearer voice with the noise removed.

[1531] Examples of specific examples and prompts

[1532] For example, suppose a user has the following conversation:

[1533] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1534] The server receives this conversation and converts it into the following text data using speech recognition software:

[1535] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1536] Next, we use a generative AI model to remove noise as follows:

[1537] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[1538] Finally, the server converts the cleaned text data back into audio data, generates processed audio, and sends it to the terminal, which plays the processed audio.

[1539] An example of a prompt sentence might be:

[1540] "Detect distinctive phrases in user conversations and remove them as noise, such as 'um' or 'Well then.'"

[1541] By inputting this prompt into a generative AI model, a process is performed to remove noise from the user's conversation.

[1542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1543] Step 1:

[1544] A user initiates a conversation.

[1545] The user begins speaking to the system, for example, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[1546] Step 2:

[1547] The device records the user's voice and sends it to the server.

[1548] The device records the user's voice using a built-in microphone. The recorded voice data is sent to the server in real time. The input is the user's voice data, and the output is the voice data sent to the server.

[1549] Step 3:

[1550] The server uses voice recognition software to convert the voice data into text data.

[1551] The server uses speech recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[1552] Step 4:

[1553] The server uses the generative AI model to analyze the text data and detect characteristic phrases.

[1554] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "Well then," are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[1555] Step 5:

[1556] The server removes the detected phrases as noise and generates clean text data.

[1557] The server removes the detected characteristic phrases as noise. For example, from the text data "Um, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Um" and "So, let's go for a walk in the afternoon" to generate clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[1558] Step 6:

[1559] The server reconverts the clean text data into audio data.

[1560] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk this afternoon" is generated. The input is cleaned text data, and the output is speech data.

[1561] Step 7:

[1562] The server transmits the generated processed voice to the terminal.

[1563] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[1564] Step 8:

[1565] The device plays the processed audio.

[1566] The device then plays back the processed audio, allowing the user to hear clear audio with the noise removed. The input is the audio data sent to the device, and the output is the audio played back from the device's speaker.

[1567] (Application example 1)

[1568] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1569] Conventional audio distribution systems have the problem that noises such as "umm" and "well then" that occur during a streamer's conversation are transmitted to the listener, degrading the quality of the audio. Streamers are also required to consciously speak to avoid the noise, making it difficult to have a natural conversation.

[1570] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed audio that does not include the removed features. This makes it possible to remove noise from conversation in real time during live streaming or audio distribution. Furthermore, by including means for generating new text based on text from which noise has been removed using a generative AI model, the broadcaster can provide clear audio to listeners while continuing a natural conversation without having to consciously avoid noise.

[1571] "User speech characteristics" refers to specific phrases and word patterns that a user uses when speaking.

[1572] "Noise" refers to phrases or words that are unnecessary or unwanted in a conversation.

[1573] "Removal" refers to the process of removing detected noise.

[1574] "Processed audio" refers to the clear audio after noise has been removed.

[1575] "Sending" refers to the process of delivering the generated processed voice to the other party.

[1576] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[1577] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[1578] "Generative AI model" refers to an algorithm or system that uses artificial intelligence to generate text or speech.

[1579] "New text" refers to the text generated by the generative AI model after noise has been removed.

[1580] "Listener" refers to a person who receives and listens to the distributed audio.

[1581] The system for carrying out this invention detects the characteristics of a user's speech, removes them as noise, generates processed speech, and transmits it. A specific embodiment of this system will be described below.

[1582] System configuration

[1583] This system uses the following hardware and software:

[1584] Hardware: Microphone, Smartphone

[1585] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (Hugging Face GPT-3 model)

[1586] Processing flow

[1587] 1. Speech recognition: The user's speech through the microphone is converted into text using the SpeechRecognition library, which allows the user's speech characteristics to be obtained as text data.

[1588] 2. Noise removal: Remove noise phrases such as "um" and "well then" from the acquired text data. This process is done by detecting specific phrases and removing them from the text.

[1589] 3. Generative AI model: Based on the noise-removed text data, a generative AI model (GPT-3) is used to generate new text. This new text will have clear content with noise removed.

[1590] 4. Generate processed speech: The new text is converted into speech and generated as processed speech, which is a clear speech with noise removed.

[1591] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive clear audio.

[1592] Specific examples

[1593] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide the stream to listeners as, "Today I'm going to talk about my new project."

[1594] Prompt Sentence Examples

[1595] Below are some example prompts to input to the generative AI model:

[1596] Remove noises like "um" and "well then" from user conversations to produce clearer voices.

[1597] By inputting this prompt into a generative AI model, clear audio with noise removed can be generated.

[1598] In this way, the present invention removes conversation noise in real time during live streaming or audio distribution, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[1599] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1600] Step 1:

[1601] The device collects the user's voice as they speak through a microphone. The device then converts the collected voice data into text data using the SpeechRecognition library. This process converts the voice data into text data. The input is voice data, and the output is text data.

[1602] Step 2:

[1603] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes any matching phrases. This process results in clean text data with the noise removed. The input is text data containing noise, and the output is text data with the noise removed.

[1604] Step 3:

[1605] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, it uses prompt sentences to give instructions to the generative AI model, which then generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new text data that has been generated.

[1606] Step 4:

[1607] The device inputs the generated new text data into a speech synthesis engine to generate processed speech. Specifically, the text data is passed to the speech synthesis engine, which outputs it as speech data. This process results in clear speech data with noise removed. The input is the generated new text data, and the output is processed speech data.

[1608] Step 5:

[1609] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[1610] Example 2

[1611] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1612] In conventional systems, the user's characteristic conversational habits and phrases remain intact, which can compromise anonymity. Furthermore, when collecting user opinions as text, manual conversion work is required, which is inefficient. This makes it difficult to completely protect the user's privacy when service personnel anonymously ask employees for user opinions.

[1613] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, means for generating processed voice that does not include the removed characteristics, means for converting the processed voice into text, and means for transmitting the text. This makes it possible to remove characteristic habits and phrases of the user and efficiently collect user opinions as text while maintaining anonymity.

[1614] "User conversation characteristics" refers to audio characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[1615] "Noise" refers to unnecessary features of a user's conversation that may compromise anonymity.

[1616] "Processed voice" refers to voice data generated by removing noise characteristic of a user's conversation and using the remaining content.

[1617] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[1618] A "prompt" is an instruction given to a generative AI model to perform a specific task.

[1619] "Means for converting to text" refers to the technology or system for analyzing audio data and converting its content into text information.

[1620] "Transmitting means" refers to the technology or system used to transmit the generated text or audio data to other devices or systems.

[1621] A "service representative" is a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[1622] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[1623] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[1624] MODE FOR CARRYING OUT THE INVENTION

[1625] The present invention provides a system for detecting characteristics of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[1626] Hardware and software used

[1627] Hardware: Servers, devices (PCs, smartphones, etc.)

[1628] Software: Generative AI models (e.g., GPT-4), audio processing software (e.g., Audacity, Adobe Audition)

[1629] System Overview

[1630] The user uses a device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server then passes the received audio file through voice processing software, which removes the user's distinctive mannerisms and phrases. The removed audio is then generated as processed voice.

[1631] Next, the server inputs a prompt sentence to the generative AI model to convert the processed speech into text. For example, the prompt sentence can be "Please convert the following speech into text: [processed speech]." The generative AI model converts the speech to text based on the prompt sentence and returns the result to the server. The server then sends the converted text to the service representative. The service representative reviews the received text and anonymously understands the user's opinion.

[1632] Specific examples

[1633] Consider a scenario where a user says, "Well, then, I don't think product A is of good quality." When the user speaks into the device's microphone, the device records this speech and saves it as a digital audio file in WAV or MP3 format. The file name will include the date and time, such as "user_input_20231010.wav."

[1634] The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload." The server passes the received audio file to its voice processing software, which removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of Product A is good."

[1635] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates text saying, "I think the quality of product A is not good," and returns it to the server. The server then sends this text to the service representative via email or chat app. The transmitted content is, "User's opinion: I think the quality of product A is not good."

[1636] The service representative reviews the received text and anonymously captures the user's opinion, which removes the user's characteristic habits and phrases.

[1637] This system allows service personnel to collect user opinions anonymously and prevents users' characteristic habits and phrases from compromising that anonymity.

[1638] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1639] Step 1:

[1640] The user provides voice input. The user speaks into the device's microphone. For example, the user might say, "Well, then, I think the quality of product A is not good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[1641] Step 2:

[1642] The device records the audio and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's audio data, and the output is the saved digital audio file.

[1643] Step 3:

[1644] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The input is a digital audio file, and the output is the audio file sent to the server.

[1645] Step 4:

[1646] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server then passes the received audio file to audio processing software (e.g., Audacity's script function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[1647] Step 5:

[1648] The server inputs a prompt to the generative AI model to convert the processed speech into text. The server inputs the prompt to the generative AI model (e.g., GPT-4) as "Please convert the following speech into text: [processed speech]". The processed speech is attached to the prompt as binary data. The input is the processed speech and the prompt, and the output is the prompt sent to the generative AI model.

[1649] Step 6:

[1650] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and processed speech, and the output is the generated text.

[1651] Step 7:

[1652] The server sends the converted text to the service representative. The server then sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[1653] Step 8:

[1654] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This allows for opinions that have been removed of the user's characteristic habits and phrases. The input is the text sent to the service representative, and the output is the confirmed user opinion.

[1655] (Application example 2)

[1656] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1657] Conventional opinion collection systems leave users' characteristic habits and phrases intact, which can compromise anonymity. It is also difficult to collect opinions in real time, making it difficult to efficiently gather feedback. This makes it difficult for users to provide honest opinions, and there is the issue of how difficult it is to obtain the information needed to improve the service.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1659] In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, and means for generating processed speech that does not include the removed features, thereby enabling efficient collection of opinions in real time while maintaining user anonymity.

[1660] "User" means any person or entity that uses the System to provide input.

[1661] "Conversational features" are individually identifiable elements such as habits, phrases, and intonation contained in a user's speech.

[1662] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[1663] "Removal" refers to the process of removing detected noise.

[1664] "Processed audio" is audio data from which noise has been removed.

[1665] "Send" is the act of transferring the processed audio to a designated recipient.

[1666] A "system" is a set of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[1667] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can also connect to the Internet and run applications.

[1668] An "application" is a software program that runs on a smartphone.

[1669] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[1670] "Real-time" refers to data processing and analysis occurring immediately.

[1671] "Security services" are services aimed at protecting information and ensuring anonymity.

[1672] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[1673] The system for implementing this invention executes a series of processes: detecting the characteristics of a user's speech, removing them as noise, generating processed speech, and transmitting it. Specifically, this is realized using an application installed on a smartphone.

[1674] Hardware and software used

[1675] Hardware: Smartphone

[1676] Software: Python, speech_recognition library, gTTS library

[1677] System Operation

[1678] 1. Speech Recognition: An application installed on a smartphone collects the user's speech, which is then converted into text using the speech_recognition library. This process uses a generative AI model to analyze the user's speech in real time.

[1679] 2. Text processing: The converted text is then stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "Well then," etc.).

[1680] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, which is a modified voice that removes the user's characteristic mannerisms and phrases.

[1681] 4. Transmission: The generated processed voice is transmitted to security service personnel, allowing them to collect opinions while maintaining the user's anonymity.

[1682] Specific examples

[1683] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[1684] Prompt Sentence Examples

[1685] Example prompts to input to a generative AI model:

[1686] Convert your users' speech to text and remove quirks and phrases to create an anonymous audio recording.

[1687] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[1688] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1689] Step 1:

[1690] The user launches the application installed on their smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[1691] Step 2:

[1692] The device converts collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze user speech in real time. Input: Collected voice data. Output: Converted text data.

[1693] Step 3:

[1694] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[1695] Step 4:

[1696] The device uses the gTTS library to convert the noise-removed text data back into speech. This speech is a processed speech that removes the user's characteristic habits and phrases. Input: Text data with noise removed. Output: Processed speech data.

[1697] Step 5:

[1698] The device sends the generated processed voice data to the security service staff. This allows the staff to collect opinions while maintaining the anonymity of the user. Input: Processed voice data. Output: Processed voice data sent to the staff.

[1699] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1700] "Example 1"

[1701] As one embodiment of the present invention, a system incorporating an emotion engine is provided. The system detects features of a user's speech, removes the features as noise, generates processed speech without the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the feature removal based on the emotions. For example, if the user is perceived as angry, the emotion engine may emphasize the angry features. Also, if the user is perceived as happy, the emotion engine may emphasize the happy features. This allows the user's emotions to be conveyed more accurately.

[1702] "Example 2"

[1703] The emotion engine also adjusts the generation of processed voices based on the user's emotion. For example, if the user feels sad, the emotion engine may emphasize the sadness characteristics. If the user feels surprised, the emotion engine may emphasize the surprise characteristics. This allows the user's emotion to be conveyed more accurately.

[1704] The processing flow of each embodiment will be described below.

[1705] "Example 1"

[1706] Step 1: Input the user's conversation into the emotion engine.

[1707] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1708] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[1709] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[1710] Step 5: Send the generated processed audio.

[1711] "Example 2"

[1712] Step 1: Input the user's conversation into the emotion engine.

[1713] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1714] Step 3: Adjust the generated processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[1715] Step 4: Generate processed audio based on the emotion-based adjustment results.

[1716] Step 5: Send the generated processed audio.

[1717] Example 1

[1718] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1719] Conventional voice analysis systems have had problems in effectively removing noises such as "umm" and "well then" that are included in user conversations, and also in not being able to properly reflect the user's emotions. As a result, the user's intentions and emotions are often not conveyed accurately.

[1720] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes means for detecting features of a user's conversation, means for removing the features as noise, means for generating processed voice that does not include the removed features, means for transmitting the processed voice, and means for analyzing the user's emotions and adjusting the removal of features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively removing noise from the user's conversation.

[1721] "User" refers to an individual or entity that uses the system to engage in conversations.

[1722] "Conversational features" refer to specific phrases and word patterns contained in a user's speech.

[1723] "Noise" refers to unnecessary phrases and words that disrupt the flow of conversation.

[1724] "Removal" refers to the process of removing detected noise from audio data.

[1725] "Processed audio" refers to audio data after noise has been removed.

[1726] "Transmit" refers to transferring the generated processed audio to another device or system.

[1727] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[1728] "Adjustment" refers to changing the noise removal method based on the results of sentiment analysis.

[1729] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases and words.

[1730] The present invention is a system that analyzes a user's conversation in real time, removes noise, and generates processed voice that reflects emotions. Specific embodiments of this system will be described below.

[1731] Hardware and software used

[1732] Hardware: Microphone, speaker, device (smartphone, tablet, PC)

[1733] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[1734] System configuration

[1735] Subject: Terminal

[1736] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[1737] Subject: Server

[1738] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects and removes noises such as "um" and "well then" from the text data. After the deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of removing the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, the "um" may not be removed to emphasize the anger.

[1739] The server converts the noise-removed text data back into audio data using text-to-speech synthesis technology, and the processed audio is sent from the server to the device.

[1740] Subject: Terminal

[1741] The device then plays the processed voice to the user through a speaker, allowing the user to hear the voice with noise removed and emotion reflected.

[1742] Specific examples

[1743] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. The emotion engine analyzes the user's emotions, and if it determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying, "Hello, it's a nice day today," is generated and sent to the device. The device then plays this processed voice back to the user.

[1744] Prompt Sentence Examples

[1745] "Detect characteristic phrases such as 'um' and 'well then' from the user's conversation and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, emphasize the angry features, and if the user is happy, emphasize the happy features."

[1746] The above is a specific embodiment for carrying out the present invention.

[1747] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1748] Step 1: Collecting audio data

[1749] Subject: Terminal

[1750] Input: User conversation

[1751] Specific operation: When the user starts speaking, the microphone on the device will automatically collect the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data will be recorded on the device.

[1752] Output: Collected audio data

[1753] Step 2: Sending audio data

[1754] Subject: Terminal

[1755] Input: Collected audio data

[1756] How it works: The device sends the collected voice data to the server in real time using an internet connection, such as Wi-Fi or mobile data.

[1757] Output: Audio data sent to the server

[1758] Step 3: Analyzing the audio data

[1759] Subject: Server

[1760] Input: Audio data sent to the server

[1761] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the generated text is "Hello, um, it's a nice day today."

[1762] Output: Text data

[1763] Step 4: Detect and remove noise

[1764] Subject: Server

[1765] Input: Text data

[1766] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" or "well then" from the text data. For example, the generated text would be "Hello, it's a nice day today."

[1767] Output: Text data with noise removed

[1768] Step 5: Sentiment analysis and adjustment

[1769] Subject: Server

[1770] Input: Text data with noise removed

[1771] Specific operation: The server analyzes the user's emotions using the emotion engine. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the server may not remove "um" to emphasize the angry characteristic.

[1772] Output: Text data reflecting emotions

[1773] Step 6: Generate processed audio

[1774] Subject: Server

[1775] Input: Text data that reflects emotions

[1776] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data such as "Hello, it's a nice day today" is generated.

[1777] Output: Processed audio data

[1778] Step 7: Send your processed audio

[1779] Subject: Server

[1780] Input: processed audio data

[1781] Specific operation: The server sends the generated processed audio to the device. This is done via an internet connection, such as Wi-Fi or mobile data.

[1782] Output: Processed audio data sent to the device

[1783] Step 8: Play the processed audio

[1784] Subject: Terminal

[1785] Input: Processed audio data sent to the device

[1786] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear "Hello, it's a nice day today."

[1787] Output: The processed audio played to the user

[1788] (Application example 1)

[1789] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1790] Conventional audio content distribution systems have the problem that unnecessary phrases and noise contained in the broadcaster's conversation are distributed as is, making the content difficult for listeners to hear. Also, since the broadcaster's emotions are not accurately conveyed, it is difficult to provide emotionally rich content to listeners.

[1791] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1792] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed audio that does not include the deleted features, means for transmitting the processed audio, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed audio, and means for transmitting the generated processed audio. This makes it possible to delete unnecessary phrases from the distributor's conversation and provide listeners with audio content that emphasizes emotions in real time.

[1793] "User conversation characteristics" refers to specific phrases and word patterns included when a user speaks.

[1794] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[1795] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[1796] "Generative AI" refers to a system that uses artificial intelligence technology to generate text or speech.

[1797] An "emotion engine" is a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[1798] "Audio Content Provider" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[1799] A "listener" refers to a user who receives, watches, or listens to audio content.

[1800] "Real-time" refers to processing occurring simultaneously as the user speaks.

[1801] "Adjusting feature removal" refers to dynamically changing the selection of phrases and words to be removed based on the user's sentiment.

[1802] "Adjusted processed audio" refers to audio data after emotions have been emphasized and noise removed by the emotion engine.

[1803] A system for implementing this invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed speech that emphasizes emotions. A specific embodiment of this system will be described below.

[1804] System Configuration

[1805] The system consists of the following main components:

[1806] 1. Speech Recognition Module: Converts the user's speech into text.

[1807] 2. Generative AI model: Analyzes text and removes unnecessary phrases.

[1808] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[1809] 4. Speech generation module: converts the adjusted text into speech.

[1810] 5. Communication module: Transmits the generated processed audio to the listener.

[1811] Hardware and software used

[1812] Speech Recognition Module: Uses the speech_recognition library.

[1813] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[1814] Sentiment analysis engine: Uses the transformers library pipeline to utilize sentiment analysis models.

[1815] Speech generation module: Uses a TTS (Text-to-Speech) engine to convert text into speech.

[1816] Communications module: uses a communications protocol to transmit audio data over the Internet.

[1817] Data processing and calculation

[1818] 1. Speech recognition: Obtain the user's voice data and convert it to text using the speech_recognition library.

[1819] 2. Noise removal: Using a generative AI model to remove unnecessary phrases from the text.

[1820] 3. Sentiment analysis: Using a sentiment analysis engine, we analyze user sentiment from text.

[1821] 4. Sentiment adjustment: Adjust the text based on the analyzed sentiment.

[1822] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[1823] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[1824] Specific examples

[1825] For example, if a streamer says, "Well, today I'm going to talk about a new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about a new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[1826] Prompt Sentence Examples

[1827] User Spoken Text: "Well, today I'm going to talk about a new project. So, let me start by giving you an overview."

[1828] Sentiment analysis result: "Joy"

[1829] Generated text: "Today we'll be talking about a new project. Let's start with an overview."

[1830] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[1831] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1832] Step 1:

[1833] A user speaks into a microphone to deliver audio content. The device captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[1834] Step 2:

[1835] The device sends the acquired voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data into text. The input is the voice data, and the output is the converted text data.

[1836] Step 3:

[1837] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[1838] Step 4:

[1839] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[1840] Step 5:

[1841] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, it processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with emphasized emotions.

[1842] Step 6:

[1843] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[1844] Step 7:

[1845] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data via the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[1846] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and emotionally-emphasized voice content can be provided to the listener.

[1847] Example 2

[1848] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1849] Conventional systems have difficulty not only removing the characteristics of a user's speech but also generating speech that accurately reflects the user's emotions. Furthermore, when collecting user opinions anonymously, there is a risk that the user's distinctive habits and phrases may remain, compromising anonymity. Furthermore, it is difficult for service personnel to accurately understand the user's emotions.

[1850] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1851] In this invention, the server includes means for detecting characteristics of a user's conversation, means for removing the characteristics as noise, and means for generating processed speech that does not include the removed characteristics. This makes it possible to remove the user's characteristic habits and phrases and collect user opinions while maintaining anonymity. The server also includes means for analyzing the user's emotions and means for generating processed speech that reflects the analyzed emotions. This allows the generated speech to accurately reflect the user's emotions, enabling the service representative to accurately understand the user's emotions.

[1852] "User conversation characteristics" refers to speech characteristics such as idiosyncrasies, phrases, and intonation that are specific to the user's speech.

[1853] "Noise" refers to distinctive habits and phrases in a user's conversations that may compromise anonymity.

[1854] "Processed voice" refers to voice data in which the characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[1855] "Means for analyzing emotions" refers to technologies and algorithms for detecting emotions from users' text data and voice data, and classifying and evaluating those emotions.

[1856] "Means for generating processed voice that reflects emotions" refers to technologies and algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions, and generating voice data that reflects the user's emotions.

[1857] A "service representative" is a person whose role is to collect user opinions and feedback and to improve or respond to the service based on that information.

[1858] "Anonymity" refers to a state in which a user's personal information or specific characteristics are protected from being known to third parties.

[1859] MODE FOR CARRYING OUT THE INVENTION

[1860] The present invention relates to a system for generating processed speech that reflects emotions by removing characteristics of a user's speech. Specific embodiments of this system will be described below.

[1861] System Overview

[1862] The system receives the user's voice data and converts it into text using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data, analyzes the user's emotions using an emotion engine, and finally uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[1863] Hardware and software used

[1864] Speech recognition software: Google Cloud Speech-to-Text API

[1865] Natural language processing libraries: spaCy, NLTK

[1866] Emotion engine: IBM Watson Tone Analyzer

[1867] Text-to-speech software: Amazon Polly

[1868] Specific examples

[1869] Consider the case where a user says, "Well, then, I don't think the quality of product A is good." The system works as follows.

[1870] 1. User voice input: The user speaks into the smartphone microphone, "Well, then, I don't think the quality of product A is good."

[1871] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[1872] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the speech data into text data such as, "Well, then, I don't think the quality of product A is good."

[1873] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Well, then," from the text data, generating text data such as "I don't think the quality of product A is good."

[1874] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[1875] 6. Generate processed speech: The server uses Amazon Polly to convert the text data, "I think the quality of product A is poor," into processed speech with a tone that reflects sadness.

[1876] 7. Sending the processed voice: The server sends the generated processed voice to the service representative, who can anonymously listen to the user's opinion through this processed voice.

[1877] Prompt Sentence Examples

[1878] An example of a prompt to input to a generative AI model is as follows:

[1879] User says: "Well, then, I don't think product A is of good quality."

[1880] Prompt for generative AI model: "Remove the user's utterances and generate a modified voice that reflects their emotions."

[1881] In this way, service personnel can listen to users' opinions anonymously and prevent their anonymity from being compromised by their characteristic habits or phrases. Furthermore, by processing the voice to reflect their emotions, service personnel can accurately understand the user's feelings.

[1882] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1883] Step 1:

[1884] The user inputs speech.

[1885] A user inputs voice using a device (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the device in a digital format (e.g., WAV or MP3).

[1886] Step 2:

[1887] The server receives the audio data.

[1888] The server receives the voice data sent from the device. The received voice data is stored in the server's storage. The input is the voice data from the device, and the output is the voice data stored in the server.

[1889] Step 3:

[1890] The server converts the voice data into text data.

[1891] The server uses speech recognition software (for example, Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[1892] Step 4:

[1893] The server removes characteristic habits and phrases from the text data.

[1894] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data with the characteristic habits and phrases removed. Specifically, unnecessary phrases such as "Well, then," are removed, and the text data that reads "I don't think the quality of product A is good" is generated.

[1895] Step 5:

[1896] The server analyzes the sentiment of the text data.

[1897] The server uses an emotion engine (for example, IBM Watson Tone Analyzer) to analyze the user's emotions from the text data. The input is text data from which characteristic habits and phrases have been removed, and the output is analyzed emotion data. Specifically, emotions such as "sad" are analyzed from the text data.

[1898] Step 6:

[1899] The server generates processed voice that reflects the emotion.

[1900] The server uses speech synthesis software (e.g., Amazon Polly) to generate processed speech that reflects emotions. The input is the analyzed emotion data and text data, and the output is processed speech that reflects emotions. Specifically, the text data, "I think the quality of product A is not good," is converted into processed speech with a tone that reflects sadness.

[1901] Step 7:

[1902] The server sends the processed voice to the service representative.

[1903] The server sends the generated processed voice to the service representative. The input is the processed voice that reflects the emotion, and the output is the processed voice sent to the service representative. The service representative can listen to the user's opinion anonymously through this processed voice.

[1904] (Application example 2)

[1905] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1906] Conventional anonymous reporting systems have had problems such as the possibility that a user's distinctive habits or phrases could compromise anonymity, and it is difficult to accurately convey the user's feelings. This can lead to the credibility and urgency of the report not being properly assessed.

[1907] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1908] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for detecting the user's emotions and adjusting the processed voice based on the emotions, and means for transmitting the adjusted processed voice, thereby enabling anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[1909] "Characteristics of a user's conversation" refers to characteristics and habits of the user's voice when speaking, specific phrases, and the like.

[1910] "Removing noise" refers to a process of removing the characteristics of a user's speech from the voice data.

[1911] "Processed voice" refers to voice data generated after removing features of the user's speech.

[1912] "Means for transmitting" refers to a function for transmitting the generated processed voice to another device or server.

[1913] "Detecting emotions" refers to the process of analyzing emotions from the content of a use...

Claims

1. A means for detecting characteristics of a user's conversation using a generative AI model; A deletion means for deleting the features included in the conversation as noise, which is a characteristic habit or phrase that may compromise the anonymity of the user, by inputting a specific prompt sentence into a generative AI model; a means for generating a processed voice that does not include the features deleted as noise as a clear voice that does not include the noise; means for transmitting the processed audio to a listener who hears the conversation; an emotion engine that recognizes the emotion of the user; and means for dynamically changing a selection of phrases and words to be deleted from the conversation based on the emotion; The system, wherein the dynamic changing means is configured to emphasize the emotional characteristics of the user by adjusting the selection of phrases and words to be deleted from the conversation based on the results of analyzing the user's emotions.

2. The system according to claim 1 , wherein the deletion means deletes the features included in the conversation during live streaming over the Internet as noise.

Images