System

Abstract

To provide a system which transmits user's emotion with improved accuracy.SOLUTION: A system comprises: means for detecting a feature of user's conversation using a generative AI model; deletion means for deleting the feature contained in the conversation as a noise, by inputting a specific prompt text into the generative AI model; means for generating processed sound not containing the feature removed as a noise, as noiseless clear sound; means for transmitting the processed sound to a listener listening to the conversation; and means for dynamically changing a phrase and a word to be deleted from the conversation based on an emotion engine recognizing user's emotion and the emotion.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] There are characteristic habits and phrases in users' conversations, which can become noise and hinder the transmission of information. In particular, when a service representative wants to ask employees anonymously for users' opinions, the characteristic habits and phrases of the user can damage the anonymity. [Means for solving the problem]

[0005] The system disclosed herein includes a means for detecting characteristics of a user's conversation using a generative AI model, a deletion means for removing the characteristics included in the conversation as noise by inputting a specific prompt sentence into the generative AI model, a means for generating processed audio that does not include the characteristics removed as noise as clear audio that does not include the noise, a means for transmitting the processed audio to a listener who listens to the conversation, an emotion engine that recognizes the user's emotions, and a means for dynamically changing the selection of phrases and words to be deleted from the conversation based on the emotions. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The present invention is a system that uses a generation AI to analyze a user's conversation in real time. This system detects the characteristics of the user's conversation, such as phrases such as "umm" and "well then," and deletes them as noise. The deletion is performed the moment the generation AI detects a characteristic phrase, and as a result, a processed voice that does not contain the characteristics is generated.

[0029] The generated processed voice is sent to the other party through the system.

[0030] "Example 2"

[0031] As a specific example of use, there is a case where a service representative wants to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the characteristics of the user's conversation. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask the user's opinion and prevents the user's characteristic habits and phrases from compromising the anonymity.

[0032] The process flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[0035] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[0036] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[0037] Step 4: A processed voice that does not contain the features is generated. This processed voice is sent to the other party through the system.

[0038] "Example 2"

[0039] Step 1: When a service representative wants to anonymously ask a user for feedback, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[0040] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[0041] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[0042] Step 4: A processed voice that does not include the characteristics is generated and sent to the service representative, allowing the service representative to listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from compromising the anonymity.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0045] Conventional voice communication systems had the problem that characteristic phrases such as "umm" and "well then" included in a user's conversation would remain as noise, making the voice difficult for the other person to hear. In addition, manually removing this noise required a lot of effort, making it impractical for real-time conversations. Furthermore, when collecting user opinions anonymously, it was difficult to accurately grasp their opinions due to the inclusion of noise.

[0046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0047] In this invention, the server includes a means for detecting a feature of a user's conversation, a means for deleting the feature as noise, and a means for generating a processed voice that does not include the deleted feature. This makes it possible to delete noise from the user's conversation in real time and generate a clear voice. The server includes a means for detecting a feature of a user's conversation using a generative AI model, a deletion means for deleting the feature included in the conversation as noise by inputting a specific prompt sentence into the generative AI model, a means for generating a processed voice that does not include the feature deleted as noise as a clear voice that does not include the noise, a means for transmitting the processed voice to a listener who listens to the conversation, an emotion engine that recognizes the emotion of the user, and a means for dynamically changing the selection of phrases and words to be deleted from the conversation based on the emotion. The deletion means deletes the feature included in the conversation during live streaming over the Internet as noise. The noise is a characteristic habit or phrase that may damage the anonymity of the user in the conversation.

[0048] "User" refers to an individual or entity that uses the system to engage in conversation.

[0049] "Conversational features" refer to specific phrases or words such as "umm" or "well then" that are included in the user's speech.

[0050] "Noise" refers to unnecessary phrases and words that are detected as characteristic of a conversation and are targeted for removal.

[0051] "Processed audio" refers to clear audio data after noise has been removed.

[0052] "Server" refers to a computer system for processing and analyzing voice data.

[0053] "Terminal" refers to a device used by a user that records and plays audio.

[0054] "Speech recognition software" refers to a program for converting voice data into text data.

[0055] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[0056] "Text data" refers to textual information converted by voice recognition software.

[0057] "Clean text data" refers to text data after noise has been removed.

[0058] "Voice data" refers to a digital representation of a user's speech.

[0059] "Real-time" refers to processing occurring at the same time that a user conversation is taking place.

[0060] The present invention relates to a system for analyzing a user's conversation in real time and removing noise. A specific embodiment of this system will be described below.

[0061] System Configuration

[0062] Hardware

[0063] Server: Uses computer systems equipped with high-performance CPUs and GPUs, which allow for large amounts of data processing and running generative AI models.

[0064] Terminal: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[0065] software

[0066] Speech recognition software: Use a common speech recognition API (e.g., Google® Speech-to-Text API) to convert voice data into text data.

[0067] Generative AI models: Use generative AI models (e.g., GPT-3(R) from OpenAI(R)) to analyze user conversations and detect characteristic phrases.

[0068] System Operation

[0069] Acquiring and sending audio data

[0070] When a user starts a conversation, the device will record the user's voice using the built-in microphone, and the recorded voice data will be sent to the server in real time.

[0071] Converting audio data to text

[0072] The server uses voice recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted into text data as is.

[0073] Analysis and noise removal of text data

[0074] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then" are detected. These phrases are removed as noise, and clean text data is generated.

[0075] Speech conversion of clean text data

[0076] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[0077] Sending and playing processed audio

[0078] The generated processed voice is transmitted from the server to the terminal, which then plays back the received processed voice, allowing the user to hear clear voice with the noise removed.

[0079] Examples of concrete examples and prompts

[0080] For example, assume that a user has the following conversation:

[0081] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0082] The server receives this conversation and converts it into the following text data using voice recognition software:

[0083] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0084] Next, we use a generative AI model to remove noise as follows:

[0085] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[0086] Finally, the server reconverts the cleaned text data into voice data, generates processed voice, and sends it to the terminal, which plays back the processed voice.

[0087] An example of a prompt sentence might be:

[0088] "Detect characteristic phrases in user conversations and remove them as noise, such as 'um' or 'well then.'"

[0089] This prompt is then fed into a generative AI model, which then processes the user's conversation to remove noise.

[0090] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0091] Step 1:

[0092] A user initiates a conversation.

[0093] The user starts speaking to the system, for example saying, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[0094] Step 2:

[0095] The device records the user's voice and transmits it to the server.

[0096] The terminal records the user's voice using a built-in microphone. The recorded voice data is transmitted to the server in real time. The input is the user's voice data, and the output is the voice data transmitted to the server.

[0097] Step 3:

[0098] The server uses voice recognition software to convert the voice data into text data.

[0099] The server uses voice recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[0100] Step 4:

[0101] The server uses a generative AI model to analyze the text data and detect characteristic phrases.

[0102] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "well then" are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[0103] Step 5:

[0104] The server removes the detected phrases as noise and generates clean text data.

[0105] The server removes the detected characteristic phrases as noise. For example, from the text data "Well, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Well" and "Well," and generates clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[0106] Step 6:

[0107] The server reconverts the clean text data into audio data.

[0108] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk in the afternoon" is generated. The input is the cleaned text data, and the output is speech data.

[0109] Step 7:

[0110] The server transmits the generated processed voice to the terminal.

[0111] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[0112] Step 8:

[0113] The device plays the processed audio.

[0114] The terminal plays the processed voice received, and the user can hear clear voice with the noise removed. The input is the voice data sent to the terminal, and the output is the voice played from the terminal speaker.

[0115] (Application example 1)

[0116] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] In conventional voice distribution systems, noises such as "umm" and "well then" that occur during the streamer's conversation are transmitted to the listener, degrading the quality of the voice. In addition, streamers are required to consciously speak to avoid noise, which makes it difficult to have a natural conversation.

[0118] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete noise from conversation in real time during live streaming or voice distribution. Furthermore, by including a means for generating new text based on text from which noise has been deleted using a generative AI model, it is possible for the distributor to provide clear voice to listeners while continuing a natural conversation without consciously avoiding noise.

[0119] "User speech characteristics" refers to the particular phrases and word patterns that a user uses when speaking.

[0120] "Noise" refers to phrases or words that are unnecessary or unwanted in a conversation.

[0121] "Deletion" refers to the process of removing detected noise.

[0122] "Processed audio" refers to the clear audio after noise has been removed.

[0123] "Sending" refers to the process of delivering the generated processed voice to the other party.

[0124] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[0125] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[0126] "Generative AI model" refers to algorithms or systems that use artificial intelligence to generate text or speech.

[0127] "New text" refers to the text generated by the generative AI model after noise has been removed.

[0128] "Listener" refers to a person who receives and listens to the distributed audio.

[0129] The system for carrying out the present invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. A specific embodiment of the system will be described below.

[0130] System Configuration

[0131] This system uses the following hardware and software:

[0132] Hardware: Microphone, Smartphone

[0133] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (GPT-3 model of Hugging Face)

[0134] Processing flow

[0135] 1. Speech recognition: The voice spoken by the user through the microphone is converted into text using the SpeechRecognition library, and the characteristics of the user's speech are obtained as text data.

[0136] 2. Noise Removal: Remove noise phrases such as "um" and "well then" from the captured text data. This process is done by detecting certain phrases and removing them from the text.

[0137] 3. Generative AI model: Based on the text data with noise removed, new text is generated using a generative AI model (GPT-3). This new text has clear content with noise removed.

[0138] 4. Generate processed speech: The new text is converted to audio and generated as processed speech, which is a clearer speech with noise removed.

[0139] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive a clear audio.

[0140] Examples

[0141] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide it to listeners as "Today I'm going to talk about my new project."

[0142] Examples of prompt statements

[0143] Below are examples of prompts to be input to the generative AI model.

[0144] Remove noises like "umm" and "well then" from the user's conversation to produce clearer voice.

[0145] By inputting this prompt sentence into a generative AI model, clear voice with noise removed can be generated.

[0146] In this way, the invention removes conversation noise in real time during live streaming or audio broadcasting, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[0147] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0148] Step 1:

[0149] The device collects the voice of the user speaking through a microphone. The device converts the collected voice data into text data using the SpeechRecognition library. Through this process, the voice data is obtained as text data. The input is voice data, and the output is text data.

[0150] Step 2:

[0151] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes matching phrases. This process results in clean text data with noise removed. The input is text data containing noise, and the output is text data with noise removed.

[0152] Step 3:

[0153] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, the device uses a prompt sentence to give instructions to the generative AI model, which generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new generated text data.

[0154] Step 4:

[0155] The terminal inputs the generated new text data to a voice synthesis engine to generate processed voice. Specifically, the text data is passed to the voice synthesis engine and output as voice data. This process results in clear voice data with noise removed. The input is the generated new text data, and the output is the processed voice data.

[0156] Step 5:

[0157] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[0158] Example 2

[0159] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] In conventional systems, the characteristic habits and phrases of a user's conversation remain intact, which may compromise anonymity. In addition, when collecting user opinions as text, manual conversion work is required, which is inefficient. This means that it is difficult to completely protect the user's privacy when a service representative anonymously asks employees for user opinions.

[0161] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating processed voice that does not include the deleted features, a means for converting the processed voice into text, and a means for transmitting the text. This makes it possible to remove the user's characteristic habits and phrases and efficiently collect user opinions as text while maintaining anonymity.

[0162] "Characteristics of a user's conversation" refer to speech characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[0163] "Noise" refers to unnecessary features of a user's conversation that may undermine anonymity.

[0164] "Processed voice" refers to voice data that has been generated by removing noise characteristic of a user's conversation and using the remaining content.

[0165] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[0166] A "prompt" refers to an instruction entered into a generative AI model to cause it to perform a specific task.

[0167] "Means of converting to text" refers to technology or systems for analyzing voice data and converting its content into text information.

[0168] "Transmitting means" refers to technologies or systems for transmitting the generated text or voice data to other devices or systems.

[0169] A "service representative" refers to a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[0170] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[0171] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[0172] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0173] The present invention provides a system for detecting features of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[0174] Hardware and Software Used

[0175] Hardware: Servers, terminals (PCs, smartphones, etc.)

[0176] Software: Generative AI models (e.g., GPT-4®), audio processing software (e.g., Audacity, Adobe Audition)

[0177] System Overview

[0178] The user uses the device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server passes the received audio file through voice processing software, which removes the user's distinctive habits and phrases. The removed voice is then generated as processed voice.

[0179] Next, the server inputs a prompt sentence to the generative AI model to convert the processed voice into text. For example, the prompt sentence is "Please convert the following voice into text: [processed voice]". The generative AI model converts the voice into text based on the prompt sentence and returns the result to the server. The server sends the converted text to the service representative. The service representative checks the received text and anonymously understands the user's opinion.

[0180] Examples

[0181] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." When the user speaks into the device's microphone, the device records this voice and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav."

[0182] The device sends the saved voice file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The server passes the received voice file to the voice processing software, which deletes unnecessary parts such as "Um, well then," and generates a processed voice that says, "I don't think the quality of Product A is good."

[0183] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates the text "I think the quality of product A is not good" and returns it to the server. The server sends this text to the service representative via email or chat app. The transmitted content is "User's opinion: I think the quality of product A is not good."

[0184] The service staff checks the received text and understands the user's opinion anonymously, which removes the user's characteristic habits and phrases.

[0185] This system allows service personnel to collect users' opinions anonymously and prevents users' characteristic habits or phrases from compromising that anonymity.

[0186] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0187] Step 1:

[0188] The user performs voice input. The user speaks into the device's microphone. For example, the user says, "Well, then, I don't think the quality of product A is good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[0189] Step 2:

[0190] The device records the voice and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's voice data, and the output is the saved digital audio file.

[0191] Step 3:

[0192] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request to the destination URL "https: / / example.com / upload". The input is the digital audio file, and the output is the audio file sent to the server.

[0193] Step 4:

[0194] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server passes the received audio file to audio processing software (e.g., Audacity's scripting function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[0195] Step 5:

[0196] The server inputs a prompt sentence to the generative AI model to convert the processed voice into text. The server inputs a prompt sentence to the generative AI model (e.g. GPT-4) saying, "Please convert the following voice into text: [processed voice]." The processed voice is attached to the prompt sentence as binary data. The input is the processed voice and the prompt sentence, and the output is the prompt sentence sent to the generative AI model.

[0197] Step 6:

[0198] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and the processed speech, and the output is the generated text.

[0199] Step 7:

[0200] The server sends the converted text to the service representative. The server sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User's opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[0201] Step 8:

[0202] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This makes it possible to obtain opinions from which the user's characteristic habits and phrases have been removed. The input is the text sent to the service representative, and the output is the confirmed user's opinion.

[0203] (Application example 2)

[0204] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0205] Conventional opinion collection systems could lose anonymity because users' characteristic habits and phrases remain intact. In addition, it was difficult to collect opinions in real time, making it difficult to collect feedback efficiently. This made it difficult for users to provide honest opinions, and there was an issue that it was difficult to obtain the information necessary to improve the service.

[0206] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0207] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[0208] "User" means any person or entity that uses the System to provide input.

[0209] "Conversational features" are individually identifiable elements contained in a user's speech, such as habits, phrases, and intonation.

[0210] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[0211] "Deletion" refers to the process of removing detected noise.

[0212] "Processed voice" refers to voice data from which noise has been removed.

[0213] "Send" is the act of forwarding the processed audio to a designated recipient.

[0214] A "system" is a collection of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[0215] A "smartphone" is a portable information terminal that, in addition to having the functions of a mobile phone, can also connect to the Internet and run applications.

[0216] An "application" is a software program that runs on a smartphone.

[0217] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[0218] "Real-time" refers to data processing and analysis being carried out immediately.

[0219] "Security service" is a service aimed at protecting information and ensuring anonymity.

[0220] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[0221] The system for implementing this invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. Specifically, this system is realized using an application installed on a smartphone.

[0222] Hardware and Software Used

[0223] Hardware: Smartphone

[0224] Software: Python, speech_recognition library, gTTS library

[0225] System Operation

[0226] 1. Speech Recognition: An application installed on a smartphone collects the user's voice. The collected voice is converted to text using the speech_recognition library. In this process, a generative AI model is used to analyze the user's speech in real time.

[0227] 2. Text processing: The converted text is stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "well then," etc.).

[0228] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, a processed voice that removes the user's idiosyncrasies and phrases.

[0229] 4. Transmission: The processed voice is transmitted to a security service officer, who can collect opinions while maintaining the anonymity of the user.

[0230] Examples

[0231] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[0232] Examples of prompt statements

[0233] Example of a prompt to be input to a generative AI model:

[0234] Convert your users' speech to text and remove quirks and phrases to produce an anonymous audio.

[0235] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[0236] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0237] Step 1:

[0238] A user starts an application installed on a smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[0239] Step 2:

[0240] The device converts the collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze the user's conversation in real time. Input: Collected voice data. Output: Converted text data.

[0241] Step 3:

[0242] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[0243] Step 4:

[0244] The device converts the text data from which the noise has been removed back into speech using the gTTS library. This speech is a processed speech from which the user's characteristic habits and phrases have been removed. Input: Text data from which noise has been removed. Output: Processed speech data.

[0245] Step 5:

[0246] The terminal transmits the generated processed voice data to the security service personnel. This allows the personnel to collect opinions while maintaining the anonymity of the users. Input: Processed voice data. Output: Processed voice data transmitted to the personnel.

[0247] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0248] "Example 1"

[0249] As an embodiment of the present invention, a system incorporating an emotion engine is provided. The system detects features of a user's speech, removes the features as noise, generates processed speech without the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the removal of features based on the emotions. For example, if the user is perceived as angry, the emotion engine may emphasize the angry features. Also, if the user is perceived as happy, the emotion engine may emphasize the happy features. This allows the user's emotions to be conveyed more accurately.

[0250] "Example 2"

[0251] The emotion engine also adjusts the generation of the processed voice based on the user's emotion. For example, if the user is perceived as sad, the emotion engine may emphasize the sadness characteristic. If the user is perceived as surprised, the emotion engine may emphasize the surprise characteristic. This allows the user's emotion to be conveyed more accurately.

[0252] The process flow of each embodiment will be described below.

[0253] "Example 1"

[0254] Step 1: Input the user's conversation into the emotion engine.

[0255] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0256] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[0257] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[0258] Step 5: Send the generated processed audio.

[0259] "Example 2"

[0260] Step 1: Input the user's conversation into the emotion engine.

[0261] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0262] Step 3: Adjust the generation of processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[0263] Step 4: Generate processed audio based on the emotion-based adjustment results.

[0264] Step 5: Send the generated processed audio.

[0265] Example 1

[0266] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0267] Conventional voice analysis systems have had problems in that it is difficult to effectively remove noises such as "umm" and "well then" contained in user conversations, and they also have problems in that they are unable to properly reflect the user's emotions. This often results in the user's intentions and emotions not being conveyed accurately.

[0268] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating a processed voice that does not include the deleted features, a means for transmitting the processed voice, and a means for analyzing the user's emotions and adjusting the deletion of the features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively deleting noise from the user's conversation.

[0269] "User" refers to an individual or entity that uses the system to engage in conversation.

[0270] "Conversational features" refer to specific phrases or word patterns contained in a user's speech.

[0271] "Noise" refers to unnecessary phrases or words that disrupt the flow of conversation.

[0272] "Removal" refers to the process of removing detected noise from the audio data.

[0273] "Processed audio" refers to audio data after noise has been removed.

[0274] "Transmit" refers to transferring the generated processed audio to another device or system.

[0275] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[0276] "Adjusting" refers to changing the method of noise removal based on the results of sentiment analysis.

[0277] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases or words.

[0278] The present invention is a system for analyzing a user's conversation in real time, removing noise, and generating processed voice that reflects emotions. A specific embodiment of this system will be described below.

[0279] Hardware and Software Used

[0280] Hardware: Microphones, speakers, devices (smartphones, tablets, PCs)

[0281] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[0282] System Configuration

[0283] Subject: Terminal

[0284] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[0285] Subject: Server

[0286] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects noise such as "um" and "well then" from the text data and deletes them. After deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of deleting the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, "um" may not be deleted in order to emphasize the anger characteristic.

[0287] The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology, and the generated processed voice is sent from the server to the terminal.

[0288] Subject: Terminal

[0289] The terminal reproduces the received processed voice through a speaker to the user, so that the user can hear the voice with noise removed and emotion reflected.

[0290] Examples

[0291] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. If the emotion engine analyzes the user's emotions and determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying "Hello, it's a nice day today" is generated and sent to the device. The device plays this processed voice to the user.

[0292] Examples of prompt statements

[0293] "Detect characteristic phrases such as 'um' and 'well then' from the user's speech and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, highlight the angry feature, and if the user is happy, highlight the happy feature."

[0294] The above is a specific embodiment for carrying out the present invention.

[0295] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0296] Step 1: Collect audio data

[0297] Subject: Terminal

[0298] Input: User conversation

[0299] Specific operation: When the user starts speaking, the microphone of the terminal automatically collects the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data is recorded on the terminal.

[0300] Output: Collected audio data

[0301] Step 2: Sending audio data

[0302] Subject: Terminal

[0303] Input: Collected audio data

[0304] How it works: The device transmits the collected voice data to the server in real time via an Internet connection, for example, Wi-Fi or mobile data.

[0305] Output: Audio data sent to the server

[0306] Step 3: Analyze the audio data

[0307] Subject: Server

[0308] Input: Audio data sent to the server

[0309] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the text generated is "Hello, um, it's a nice day today."

[0310] Output: Text data

[0311] Step 4: Detect and remove noise

[0312] Subject: Server

[0313] Input: Text data

[0314] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" and "well then" from text data. For example, the generated text is "Hello, it's a nice day today."

[0315] Output: Text data with noise removed

[0316] Step 5: Sentiment analysis and adjustment

[0317] Subject: Server

[0318] Input: Text data with noise removed

[0319] Specific operation: The server uses the emotion engine to analyze the user's emotions. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the "um" may not be removed to emphasize the angry characteristic.

[0320] Output: Text data reflecting emotions

[0321] Step 6: Generate processed audio

[0322] Subject: Server

[0323] Input: Text data reflecting emotions

[0324] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data saying "Hello, it's a nice day today" is generated.

[0325] Output: Processed audio data

[0326] Step 7: Send your processed audio

[0327] Subject: Server

[0328] Input: Processed audio data

[0329] Specific operation: The server transmits the generated processed voice to the terminal. The transmission is performed using an Internet connection, for example, using Wi-Fi or mobile data communication.

[0330] Output: Processed audio data sent to the device

[0331] Step 8: Play the processed audio

[0332] Subject: Terminal

[0333] Input: Processed voice data sent to the terminal

[0334] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear the voice saying "Hello, it's a nice day today."

[0335] Output: The processed audio played to the user

[0336] (Application example 1)

[0337] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] Conventional audio content delivery systems have the problem that unnecessary phrases and noises contained in the distributor's conversation are delivered as is, making the content difficult for listeners to hear. In addition, the distributor's emotions are not accurately conveyed, making it difficult to provide emotionally rich content to listeners.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0340] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for transmitting the processed voice, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed voice, and means for transmitting the generated processed voice. This makes it possible to provide listeners with audio content in real time that deletes unnecessary phrases from the distributor's conversation and emphasizes emotions.

[0341] "User conversation characteristics" refers to specific phrases or word patterns contained when a user speaks.

[0342] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[0343] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[0344] "Generative AI" refers to a system that uses artificial intelligence technology to generate text and speech.

[0345] An "emotion engine" refers to a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[0346] "Audio content distributor" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[0347] A "listener" refers to a user who receives, views, or listens to audio content.

[0348] "Real-time" refers to processing occurring simultaneously as the user's speech is being spoken.

[0349] "Tuning feature removal" refers to dynamically changing the selection of phrases or words to be removed based on the user's sentiment.

[0350] "Adjusted processed voice" refers to voice data after emotions have been enhanced and noise has been removed by the emotion engine.

[0351] A system for carrying out the present invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed voice with emphasized emotions. A specific embodiment of this system will be described below.

[0352] System Configuration

[0353] The system consists of the following main components:

[0354] 1. Speech Recognition Module: Converts the user's speech into text.

[0355] 2. Generative AI model: Analyzes the text and removes unnecessary phrases.

[0356] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[0357] 4. Speech generation module: Converts the tuned text into speech.

[0358] 5. Communication module: Transmits the generated processed audio to the listener.

[0359] Hardware and Software Used

[0360] Speech recognition module: Uses the speech_recognition library.

[0361] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[0362] Sentiment analysis engine: Uses the transformers library pipeline to leverage sentiment analysis models.

[0363] Speech generation module: Uses a Text-to-Speech (TTS) engine to convert text into speech.

[0364] Communications Module: Uses a communications protocol to transmit voice data over the Internet.

[0365] Data processing and calculation

[0366] 1. Speech recognition: Obtain user voice data and convert it to text using the speech_recognition library.

[0367] 2. Noise removal: Use a generative AI model to remove unnecessary phrases from the text.

[0368] 3. Sentiment Analysis: Using a sentiment analysis engine, we analyze user sentiment from the text.

[0369] 4. Sentiment Adjustment: Adjust the text based on the analyzed sentiment.

[0370] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[0371] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[0372] Examples

[0373] For example, if a streamer says, "Well, today I'm going to talk about my new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about my new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[0374] Examples of prompt statements

[0375] User spoken text: "Well, today I'm going to be talking about a new project. So, let me start by giving you an overview."

[0376] Sentiment Analysis Result: "Joy"

[0377] Generated text: "Today I'm going to talk about a new project. Let me start by giving you an overview."

[0378] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[0379] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0380] Step 1:

[0381] A user speaks into a microphone to deliver audio content. The terminal captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[0382] Step 2:

[0383] The device sends the captured voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data to text. The input is the voice data, and the output is the converted text data.

[0384] Step 3:

[0385] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[0386] Step 4:

[0387] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[0388] Step 5:

[0389] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, the server processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with the emotion emphasized.

[0390] Step 6:

[0391] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[0392] Step 7:

[0393] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data over the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[0394] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and voice content with emphasized emotions can be provided to the listener.

[0395] Example 2

[0396] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0397] In conventional systems, it was difficult not only to remove the characteristics of a user's conversation, but also to generate speech that accurately reflected the user's emotions. In addition, when collecting user opinions anonymously, there was a possibility that anonymity would be lost if the user's characteristic habits and phrases remained. Furthermore, it was difficult for service personnel to accurately understand the user's emotions.

[0398] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0399] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete the user's characteristic habits and phrases and collect the user's opinions while maintaining anonymity. The server also includes a means for analyzing the user's emotions and a means for generating processed voice that reflects the analyzed emotions. This makes it possible to generate a voice that accurately reflects the user's emotions, allowing the service staff to accurately understand the user's emotions.

[0400] "User conversation characteristics" refers to speech characteristics such as idiosyncrasies, phrases, and intonation that are specific to the user's speech.

[0401] "Noise" refers to distinctive habits and phrases in a user's conversations that may undermine anonymity.

[0402] "Processed voice" refers to voice data in which characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[0403] "Means of emotion analysis" refers to technologies and algorithms for detecting emotions from a user's text data or voice data and classifying and evaluating those emotions.

[0404] "Means for generating processed voice that reflects emotions" refers to technologies or algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions and generating voice data that reflects the user's emotions.

[0405] A "service representative" is a person whose role is to collect user opinions and feedback and to improve and respond to the service based on that.

[0406] "Anonymity" refers to a state in which a user's personal information and specific characteristics are protected from being known to third parties.

[0407] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0408] The present invention relates to a system for generating processed speech that reflects emotions by removing features of a user's speech. A specific embodiment of this system will be described below.

[0409] System Overview

[0410] The system receives the user's voice data and converts it into text data using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data and uses an emotion engine to analyze the user's emotions. Finally, it uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[0411] Hardware and Software Used

[0412] Speech recognition software: Google Cloud Speech-to-Text API

[0413] Natural Language Processing Libraries: spaCy, NLTK

[0414] Emotion Engine: IBM Watson(R) Tone Analyzer

[0415] Text-to-speech software: Amazon Polly

[0416] Examples

[0417] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." This system works as follows.

[0418] 1. User voice input: The user speaks into the microphone on their smartphone, saying, "Well, you see, I don't think the quality of product A is good."

[0419] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[0420] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as, "Well, then, I don't think the quality of Product A is good."

[0421] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Um, so..." from the text data, generating text data such as "I don't think the quality of Product A is good."

[0422] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[0423] 6. Generate processed voice: The server uses Amazon Polly to convert the text data, “I think the quality of product A is poor” into a processed voice with a tone that reflects sadness.

[0424] 7. Sending the processed voice: The server sends the generated processed voice to the service representative. The service representative can listen to the user's opinion anonymously through the processed voice.

[0425] Examples of prompt statements

[0426] Here is an example of a prompt to be input to the generative AI model:

[0427] User says: "Well, then, I don't think product A is of good quality."

[0428] Prompt for generative AI model: "Remove idiosyncrasies from user's speech and generate a processed voice that reflects the user's emotions."

[0429] In this way, the service representative can listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from damaging the anonymity. In addition, the processed voice reflecting the user's emotions makes it possible to accurately understand the user's emotions.

[0430] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0431] Step 1:

[0432] The user inputs speech.

[0433] A user inputs voice using a terminal (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the terminal in a digital format (e.g., WAV or MP3).

[0434] Step 2:

[0435] The server receives the audio data.

[0436] The server receives the voice data sent from the terminal. The received voice data is stored in the storage of the server. The input is the voice data from the terminal, and the output is the voice data stored in the server.

[0437] Step 3:

[0438] The server converts the voice data into text data.

[0439] The server uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[0440] Step 4:

[0441] The server removes distinctive habits and phrases from the text data.

[0442] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data from which the characteristic habits and phrases have been removed. Specifically, unnecessary phrases such as "Well, then," are deleted, and text data such as "I don't think the quality of product A is good" is generated.

[0443] Step 5:

[0444] The server analyzes the sentiment of the text data.

[0445] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from the text data. The input is text data from which characteristic habits and phrases have been removed, and the output is the analyzed emotion data. Specifically, emotions such as "sad" are analyzed from the text data.

[0446] Step 6:

[0447] The server generates processed voice that reflects the emotion.

[0448] The server uses speech synthesis software (e.g., Amazon Polly) to generate processed speech that reflects emotions. The input is the analyzed emotion data and text data, and the output is processed speech that reflects emotions. Specifically, text data such as "I think the quality of product A is not good" is converted into processed speech with a tone that reflects sadness.

[0449] Step 7:

[0450] The server transmits the processed voice to a service representative.

[0451] The server transmits the generated processed voice to the service representative. The input is the processed voice reflecting the emotion, and the output is the processed voice transmitted to the service representative. The service representative can listen to the user's opinion anonymously through the processed voice.

[0452] (Application example 2)

[0453] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0454] Conventional anonymous reporting systems have problems in that a user's characteristic habits or phrases can undermine anonymity, and it is difficult to accurately convey the user's feelings. This can lead to the credibility and urgency of the report not being properly evaluated.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0456] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating a processed voice that does not include the deleted features, means for detecting the user's emotions and adjusting the processed voice based on the emotions, and means for transmitting the adjusted processed voice, thereby enabling anonymous reporting that accurately conveys emotions while maintaining the anonymity of the user.

[0457] "Characteristics of a user's conversation" refers to characteristics and habits of the user's voice when speaking, specific phrases, and the like.

[0458] "Removing as noise" refers to the process of removing characteristics of a user's speech from the voice data.

[0459] "Processed voice" refers to voice data that is generated after removing features of the user's speech.

[0460] "Means for transmitting" refers to a function for transmitting the generated processed voice to another device or server.

[0461] "Detecting emotions" refers to the process of analyzing emotions from the content of a user's speech and the tone of their voice.

[0462] "Adjusting the processed voice based on emotion" refers to the process of processing voice data to reflect detected emotion.

[0463] "Making an anonymous report" refers to the act of sending the report content while concealing the user's personal information and characteristics.

[0464] A system for implementing this invention detects characteristics of a user's speech, deletes the characteristics as noise, generates a processed voice that does not include the deleted characteristics, detects the user's emotions and adjusts the processed voice based on the emotions, and finally transmits the adjusted processed voice.

[0465] Hardware and Software Used

[0466] Hardware: Smartphone

[0467] Software: Python, speech_recognition library, pydub library, librosa library, soundfile library

[0468] Processing flow

[0469] 1. Speech recognition: The smartphone microphone is used to record the user's voice, and the speech_recognition library is used to extract text from the voice data, thereby obtaining the user's spoken content as text data.

[0470] 2. Voice processing: Using the librosa library, we remove the user's characteristic habits and phrases from the voice data. Specifically, we analyze the characteristics of the voice and remove the characteristic parts as noise.

[0471] 3. Emotion detection: Using the librosa library, we analyze the tone and pitch of the voice data to detect the user's emotions. For example, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0472] 4. Emotion-based speech adjustment: Reprocessing the speech data based on the detected emotion, for example slowing down the speech to emphasize the emotion of sadness or raising the pitch of the speech to emphasize the emotion of surprise.

[0473] 5. Audio transmission: The processed audio data is saved using the soundfile library and sent to the reporting center using the smartphone's communication function.

[0474] Examples

[0475] When a user reports, "I saw a suspicious person in my neighborhood," the system generates a processed voice saying, "I saw a suspicious person," and sends it to the reporting center. Furthermore, if the user is feeling anxious, the system can emphasize the feeling of anxiety.

[0476] Examples of prompt statements

[0477] When a user reports "I saw a suspicious person in my neighborhood," the application generates a processed voice saying "I saw a suspicious person" and sends it to the reporting center. Furthermore, if the user is feeling anxious, the application can emphasize the emotion of anxiety.

[0478] In this way, anonymous reporting that accurately conveys feelings is possible while maintaining the anonymity of the user.

[0479] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0480] Step 1:

[0481] A user records audio using the smartphone microphone. The input is the user's speech and the output is an audio file (e.g., input.wav). This audio file is used in subsequent processing steps.

[0482] Step 2:

[0483] The device uses the speech_recognition library to extract text from an audio file. The input is an audio file, and the output is text data (e.g., "I saw a suspicious person in my neighborhood."). This text data is used to obtain the user's spoken content as text information.

[0484] Step 3:

[0485] The device uses the librosa library to remove the user's characteristic habits and phrases from the voice data. The input is an audio file, and the output is the voice data with the characteristics removed. Specifically, the characteristics of the voice are analyzed, and the characteristic parts are removed as noise.

[0486] Step 4:

[0487] The device uses the librosa library to analyze the tone and pitch of the voice data to detect the user's emotions. The input is an audio file, and the output is the detected emotion (e.g., "anxiety"). Specifically, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0488] Step 5:

[0489] The device reprocesses the speech data based on the detected emotion: the input is the feature-removed speech data and the detected emotion, and the output is processed speech with the emotion emphasized, for example slowing the speech to emphasize the emotion of sadness or raising the pitch of the speech to emphasize the emotion of surprise.

[0490] Step 6:

[0491] The device uses the soundfile library to save the processed voice data. The input is the processed voice with emphasis on emotions, and the output is the saved audio file (e.g. output.wav). This audio file is used to send to the reporting center.

[0492] Step 7:

[0493] The device uses the smartphone's communication function to send the processed voice file to the reporting center. The input is the saved voice file, and the output is the voice data sent to the reporting center. This allows anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[0494] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0495] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0496] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0497] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0498] [Second embodiment]

[0499] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0500] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0501] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0502] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0503] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0504] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0505] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0506] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0507] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0508] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0509] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0510] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0511] "Example 1"

[0512] The present invention is a system that uses a generation AI to analyze a user's conversation in real time. This system detects features of the user's conversation, such as phrases such as "umm" and "well then," and deletes them as noise. The deletion is performed the moment the generation AI detects a characteristic phrase, and as a result, a processed voice that does not contain the features is generated. The generated processed voice is transmitted to the other party through the system.

[0513] "Example 2"

[0514] As a specific example of use, there is a case where a service representative wants to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the characteristics of the user's conversation. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask the user's opinion and prevents the user's characteristic habits and phrases from compromising the anonymity.

[0515] The process flow of each embodiment will be described below.

[0516] "Example 1"

[0517] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[0518] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[0519] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[0520] Step 4: A processed voice that does not contain the features is generated. This processed voice is sent to the other party through the system.

[0521] "Example 2"

[0522] Step 1: When a service representative wants to anonymously ask a user for feedback, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[0523] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[0524] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[0525] Step 4: A processed voice that does not include the characteristics is generated and sent to the service representative, allowing the service representative to listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from compromising the anonymity.

[0526] Example 1

[0527] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0528] Conventional voice communication systems had the problem that characteristic phrases such as "umm" and "well then" included in a user's conversation would remain as noise, making the voice difficult for the other person to hear. In addition, manually removing this noise required a lot of effort, making it impractical for real-time conversations. Furthermore, when collecting user opinions anonymously, it was difficult to accurately grasp their opinions due to the inclusion of noise.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0530] In this invention, the server includes a means for detecting features of a user's speech, a means for deleting the features as noise, and a means for generating processed speech that does not include the deleted features. This makes it possible to delete noise from a user's speech in real time and generate clear speech.

[0531] "User" refers to an individual or entity that uses the system to engage in conversation.

[0532] "Conversational features" refer to specific phrases or words such as "umm" or "well then" that are included in the user's speech.

[0533] "Noise" refers to unnecessary phrases and words that are detected as characteristic of a conversation and are targeted for removal.

[0534] "Processed audio" refers to clear audio data after noise has been removed.

[0535] "Server" refers to a computer system for processing and analyzing voice data.

[0536] "Terminal" refers to a device used by a user that records and plays audio.

[0537] "Speech recognition software" refers to a program for converting voice data into text data.

[0538] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[0539] "Text data" refers to textual information converted by voice recognition software.

[0540] "Clean text data" refers to text data after noise has been removed.

[0541] "Voice data" refers to a digital representation of a user's speech.

[0542] "Real-time" refers to processing occurring at the same time that a user conversation is taking place.

[0543] The present invention relates to a system for analyzing a user's conversation in real time and removing noise. A specific embodiment of this system will be described below.

[0544] System Configuration

[0545] Hardware

[0546] Server: Uses computer systems equipped with high-performance CPUs and GPUs, which allow for large amounts of data processing and running generative AI models.

[0547] Terminal: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[0548] software

[0549] Speech recognition software: Use common speech recognition APIs (e.g. Google Speech-to-Text API) to convert voice data into text data.

[0550] Generative AI models: Use generative AI models (e.g., OpenAI's GPT-3) to analyze user conversations and detect signature phrases.

[0551] System Operation

[0552] Acquiring and sending audio data

[0553] When a user starts a conversation, the device will record the user's voice using the built-in microphone, and the recorded voice data will be sent to the server in real time.

[0554] Converting audio data to text

[0555] The server uses voice recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted into text data as is.

[0556] Analysis and noise removal of text data

[0557] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then" are detected. These phrases are removed as noise, and clean text data is generated.

[0558] Speech conversion of clean text data

[0559] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[0560] Sending and playing processed audio

[0561] The generated processed voice is transmitted from the server to the terminal, which then plays back the received processed voice, allowing the user to hear clear voice with the noise removed.

[0562] Examples of concrete examples and prompts

[0563] For example, assume that a user has the following conversation:

[0564] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0565] The server receives this conversation and converts it into the following text data using voice recognition software:

[0566] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[0567] Next, we use a generative AI model to remove noise as follows:

[0568] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[0569] Finally, the server reconverts the cleaned text data into voice data, generates processed voice, and sends it to the terminal, which plays back the processed voice.

[0570] An example of a prompt sentence might be:

[0571] "Detect characteristic phrases in user conversations and remove them as noise, such as 'um' or 'well then.'"

[0572] This prompt is then fed into a generative AI model, which then processes the user's conversation to remove noise.

[0573] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0574] Step 1:

[0575] A user initiates a conversation.

[0576] The user starts speaking to the system, for example saying, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[0577] Step 2:

[0578] The device records the user's voice and transmits it to the server.

[0579] The terminal records the user's voice using a built-in microphone. The recorded voice data is transmitted to the server in real time. The input is the user's voice data, and the output is the voice data transmitted to the server.

[0580] Step 3:

[0581] The server uses voice recognition software to convert the voice data into text data.

[0582] The server uses voice recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[0583] Step 4:

[0584] The server uses a generative AI model to analyze the text data and detect characteristic phrases.

[0585] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "well then" are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[0586] Step 5:

[0587] The server removes the detected phrases as noise and generates clean text data.

[0588] The server removes the detected characteristic phrases as noise. For example, from the text data "Well, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Well" and "Well," and generates clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[0589] Step 6:

[0590] The server reconverts the clean text data into audio data.

[0591] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk in the afternoon" is generated. The input is the cleaned text data, and the output is speech data.

[0592] Step 7:

[0593] The server transmits the generated processed voice to the terminal.

[0594] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[0595] Step 8:

[0596] The device plays the processed audio.

[0597] The terminal plays the processed voice received, and the user can hear clear voice with the noise removed. The input is the voice data sent to the terminal, and the output is the voice played from the terminal speaker.

[0598] (Application example 1)

[0599] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0600] In conventional voice distribution systems, noises such as "umm" and "well then" that occur during the streamer's conversation are transmitted to the listener, degrading the quality of the voice. In addition, streamers are required to consciously speak to avoid noise, which makes it difficult to have a natural conversation.

[0601] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete noise from conversation in real time during live streaming or voice distribution. Furthermore, by including a means for generating new text based on text from which noise has been deleted using a generative AI model, it is possible for the distributor to provide clear voice to listeners while continuing a natural conversation without consciously avoiding noise.

[0602] "User speech characteristics" refers to the particular phrases and word patterns that a user uses when speaking.

[0603] "Noise" refers to phrases or words that are unnecessary or unwanted in a conversation.

[0604] "Deletion" refers to the process of removing detected noise.

[0605] "Processed audio" refers to the clear audio after noise has been removed.

[0606] "Sending" refers to the process of delivering the generated processed voice to the other party.

[0607] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[0608] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[0609] "Generative AI model" refers to algorithms or systems that use artificial intelligence to generate text or speech.

[0610] "New text" refers to the text generated by the generative AI model after noise has been removed.

[0611] "Listener" refers to a person who receives and listens to the distributed audio.

[0612] The system for carrying out the present invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. A specific embodiment of the system will be described below.

[0613] System Configuration

[0614] This system uses the following hardware and software:

[0615] Hardware: Microphone, Smartphone

[0616] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (GPT-3 model of Hugging Face)

[0617] Processing flow

[0618] 1. Speech recognition: The voice spoken by the user through the microphone is converted into text using the SpeechRecognition library, and the characteristics of the user's speech are obtained as text data.

[0619] 2. Noise Removal: Remove noise phrases such as "um" and "well then" from the captured text data. This process is done by detecting certain phrases and removing them from the text.

[0620] 3. Generative AI model: Based on the text data with noise removed, new text is generated using a generative AI model (GPT-3). This new text has clear content with noise removed.

[0621] 4. Generate processed speech: The new text is converted to audio and generated as processed speech, which is a clearer speech with noise removed.

[0622] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive a clear audio.

[0623] Examples

[0624] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide it to listeners as "Today I'm going to talk about my new project."

[0625] Examples of prompt statements

[0626] Below are examples of prompts to be input to the generative AI model.

[0627] Remove noises like "umm" and "well then" from the user's conversation to produce clearer voice.

[0628] By inputting this prompt sentence into a generative AI model, clear voice with noise removed can be generated.

[0629] In this way, the invention removes conversation noise in real time during live streaming or audio broadcasting, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[0630] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0631] Step 1:

[0632] The device collects the voice of the user speaking through a microphone. The device converts the collected voice data into text data using the SpeechRecognition library. Through this process, the voice data is obtained as text data. The input is voice data, and the output is text data.

[0633] Step 2:

[0634] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes matching phrases. This process results in clean text data with noise removed. The input is text data containing noise, and the output is text data with noise removed.

[0635] Step 3:

[0636] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, the device uses a prompt sentence to give instructions to the generative AI model, which generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new generated text data.

[0637] Step 4:

[0638] The terminal inputs the generated new text data to a voice synthesis engine to generate processed voice. Specifically, the text data is passed to the voice synthesis engine and output as voice data. This process results in clear voice data with noise removed. The input is the generated new text data, and the output is the processed voice data.

[0639] Step 5:

[0640] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[0641] Example 2

[0642] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0643] In conventional systems, the characteristic habits and phrases of a user's conversation remain intact, which may compromise anonymity. In addition, when collecting user opinions as text, manual conversion work is required, which is inefficient. This means that it is difficult to completely protect the user's privacy when a service representative anonymously asks employees for user opinions.

[0644] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating processed voice that does not include the deleted features, a means for converting the processed voice into text, and a means for transmitting the text. This makes it possible to remove the user's characteristic habits and phrases and efficiently collect user opinions as text while maintaining anonymity.

[0645] "Characteristics of a user's conversation" refer to speech characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[0646] "Noise" refers to unnecessary features of a user's conversation that may undermine anonymity.

[0647] "Processed voice" refers to voice data that has been generated by removing noise characteristic of a user's conversation and using the remaining content.

[0648] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[0649] A "prompt" refers to an instruction entered into a generative AI model to cause it to perform a specific task.

[0650] "Means of converting to text" refers to technology or systems for analyzing voice data and converting its content into text information.

[0651] "Transmitting means" refers to technologies or systems for transmitting the generated text or voice data to other devices or systems.

[0652] A "service representative" refers to a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[0653] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[0654] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[0655] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0656] The present invention provides a system for detecting features of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[0657] Hardware and Software Used

[0658] Hardware: Servers, terminals (PCs, smartphones, etc.)

[0659] Software: Generative AI models (e.g. GPT-4), audio processing software (e.g. Audacity, Adobe Audition)

[0660] System Overview

[0661] The user uses the device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server passes the received audio file through voice processing software, which removes the user's distinctive habits and phrases. The removed voice is then generated as processed voice.

[0662] Next, the server inputs a prompt sentence to the generative AI model to convert the processed voice into text. For example, the prompt sentence is "Please convert the following voice into text: [processed voice]". The generative AI model converts the voice into text based on the prompt sentence and returns the result to the server. The server sends the converted text to the service representative. The service representative checks the received text and anonymously understands the user's opinion.

[0663] Examples

[0664] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." When the user speaks into the device's microphone, the device records this voice and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav."

[0665] The device sends the saved voice file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The server passes the received voice file to the voice processing software, which deletes unnecessary parts such as "Um, well then," and generates a processed voice that says, "I don't think the quality of Product A is good."

[0666] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates the text "I think the quality of product A is not good" and returns it to the server. The server sends this text to the service representative via email or chat app. The transmitted content is "User's opinion: I think the quality of product A is not good."

[0667] The service staff checks the received text and understands the user's opinion anonymously, which removes the user's characteristic habits and phrases.

[0668] This system allows service personnel to collect users' opinions anonymously and prevents users' characteristic habits or phrases from compromising that anonymity.

[0669] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0670] Step 1:

[0671] The user performs voice input. The user speaks into the device's microphone. For example, the user says, "Well, then, I don't think the quality of product A is good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[0672] Step 2:

[0673] The device records the voice and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's voice data, and the output is the saved digital audio file.

[0674] Step 3:

[0675] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request to the destination URL "https: / / example.com / upload". The input is the digital audio file, and the output is the audio file sent to the server.

[0676] Step 4:

[0677] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server passes the received audio file to audio processing software (e.g., Audacity's scripting function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[0678] Step 5:

[0679] The server inputs a prompt sentence to the generative AI model to convert the processed voice into text. The server inputs a prompt sentence to the generative AI model (e.g. GPT-4) saying, "Please convert the following voice into text: [processed voice]." The processed voice is attached to the prompt sentence as binary data. The input is the processed voice and the prompt sentence, and the output is the prompt sentence sent to the generative AI model.

[0680] Step 6:

[0681] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and the processed speech, and the output is the generated text.

[0682] Step 7:

[0683] The server sends the converted text to the service representative. The server sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User's opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[0684] Step 8:

[0685] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This makes it possible to obtain opinions from which the user's characteristic habits and phrases have been removed. The input is the text sent to the service representative, and the output is the confirmed user's opinion.

[0686] (Application example 2)

[0687] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0688] Conventional opinion collection systems could lose anonymity because users' characteristic habits and phrases remain intact. In addition, it was difficult to collect opinions in real time, making it difficult to collect feedback efficiently. This made it difficult for users to provide honest opinions, and there was an issue that it was difficult to obtain the information necessary to improve the service.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0690] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[0691] "User" means any person or entity that uses the System to provide input.

[0692] "Conversational features" are individually identifiable elements contained in a user's speech, such as habits, phrases, and intonation.

[0693] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[0694] "Deletion" refers to the process of removing detected noise.

[0695] "Processed voice" refers to voice data from which noise has been removed.

[0696] "Send" is the act of forwarding the processed audio to a designated recipient.

[0697] A "system" is a collection of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[0698] A "smartphone" is a portable information terminal that, in addition to having the functions of a mobile phone, can also connect to the Internet and run applications.

[0699] An "application" is a software program that runs on a smartphone.

[0700] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[0701] "Real-time" refers to data processing and analysis being carried out immediately.

[0702] "Security service" is a service aimed at protecting information and ensuring anonymity.

[0703] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[0704] The system for implementing this invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. Specifically, this system is realized using an application installed on a smartphone.

[0705] Hardware and Software Used

[0706] Hardware: Smartphone

[0707] Software: Python, speech_recognition library, gTTS library

[0708] System Operation

[0709] 1. Speech Recognition: An application installed on a smartphone collects the user's voice. The collected voice is converted to text using the speech_recognition library. In this process, a generative AI model is used to analyze the user's speech in real time.

[0710] 2. Text processing: The converted text is stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "well then," etc.).

[0711] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, a processed voice that removes the user's idiosyncrasies and phrases.

[0712] 4. Transmission: The processed voice is transmitted to a security service officer, who can collect opinions while maintaining the anonymity of the user.

[0713] Examples

[0714] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[0715] Examples of prompt statements

[0716] Example of a prompt to be input to a generative AI model:

[0717] Convert your users' speech to text and remove quirks and phrases to produce an anonymous audio.

[0718] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[0719] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0720] Step 1:

[0721] A user starts an application installed on a smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[0722] Step 2:

[0723] The device converts the collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze the user's conversation in real time. Input: Collected voice data. Output: Converted text data.

[0724] Step 3:

[0725] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[0726] Step 4:

[0727] The device converts the text data from which the noise has been removed back into speech using the gTTS library. This speech is a processed speech from which the user's characteristic habits and phrases have been removed. Input: Text data from which noise has been removed. Output: Processed speech data.

[0728] Step 5:

[0729] The terminal transmits the generated processed voice data to the security service personnel. This allows the personnel to collect opinions while maintaining the anonymity of the users. Input: Processed voice data. Output: Processed voice data transmitted to the personnel.

[0730] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0731] "Example 1"

[0732] As an embodiment of the present invention, a system incorporating an emotion engine is provided. The system detects features of a user's speech, removes the features as noise, generates processed speech without the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the removal of features based on the emotions. For example, if the user is perceived as angry, the emotion engine may emphasize the angry features. Also, if the user is perceived as happy, the emotion engine may emphasize the happy features. This allows the user's emotions to be conveyed more accurately.

[0733] "Example 2"

[0734] The emotion engine also adjusts the generation of the processed voice based on the user's emotion. For example, if the user is perceived as sad, the emotion engine may emphasize the sadness characteristic. If the user is perceived as surprised, the emotion engine may emphasize the surprise characteristic. This allows the user's emotion to be conveyed more accurately.

[0735] The process flow of each embodiment will be described below.

[0736] "Example 1"

[0737] Step 1: Input the user's conversation into the emotion engine.

[0738] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0739] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[0740] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[0741] Step 5: Send the generated processed audio.

[0742] "Example 2"

[0743] Step 1: Input the user's conversation into the emotion engine.

[0744] Step 2: The emotion engine analyzes emotions from the user's conversation.

[0745] Step 3: Adjust the generation of processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[0746] Step 4: Generate processed audio based on the emotion-based adjustment results.

[0747] Step 5: Send the generated processed audio.

[0748] Example 1

[0749] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0750] Conventional voice analysis systems have had problems in that it is difficult to effectively remove noises such as "umm" and "well then" contained in user conversations, and they also have problems in that they are unable to properly reflect the user's emotions. This often results in the user's intentions and emotions not being conveyed accurately.

[0751] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating a processed voice that does not include the deleted features, a means for transmitting the processed voice, and a means for analyzing the user's emotions and adjusting the deletion of the features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively deleting noise from the user's conversation.

[0752] "User" refers to an individual or entity that uses the system to engage in conversation.

[0753] "Conversational features" refer to specific phrases or word patterns contained in a user's speech.

[0754] "Noise" refers to unnecessary phrases or words that disrupt the flow of conversation.

[0755] "Removal" refers to the process of removing detected noise from the audio data.

[0756] "Processed audio" refers to audio data after noise has been removed.

[0757] "Transmit" refers to transferring the generated processed audio to another device or system.

[0758] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[0759] "Adjusting" refers to changing the method of noise removal based on the results of sentiment analysis.

[0760] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases or words.

[0761] The present invention is a system for analyzing a user's conversation in real time, removing noise, and generating processed voice that reflects emotions. A specific embodiment of this system will be described below.

[0762] Hardware and Software Used

[0763] Hardware: Microphones, speakers, devices (smartphones, tablets, PCs)

[0764] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[0765] System Configuration

[0766] Subject: Terminal

[0767] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[0768] Subject: Server

[0769] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects noise such as "um" and "well then" from the text data and deletes them. After deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of deleting the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, "um" may not be deleted in order to emphasize the anger characteristic.

[0770] The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology, and the generated processed voice is sent from the server to the terminal.

[0771] Subject: Terminal

[0772] The terminal reproduces the received processed voice through a speaker to the user, so that the user can hear the voice with noise removed and emotion reflected.

[0773] Examples

[0774] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. If the emotion engine analyzes the user's emotions and determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying "Hello, it's a nice day today" is generated and sent to the device. The device plays this processed voice to the user.

[0775] Examples of prompt statements

[0776] "Detect characteristic phrases such as 'um' and 'well then' from the user's speech and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, highlight the angry feature, and if the user is happy, highlight the happy feature."

[0777] The above is a specific embodiment for carrying out the present invention.

[0778] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0779] Step 1: Collect audio data

[0780] Subject: Terminal

[0781] Input: User conversation

[0782] Specific operation: When the user starts speaking, the microphone of the terminal automatically collects the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data is recorded on the terminal.

[0783] Output: Collected audio data

[0784] Step 2: Sending audio data

[0785] Subject: Terminal

[0786] Input: Collected audio data

[0787] How it works: The device transmits the collected voice data to the server in real time via an Internet connection, for example, Wi-Fi or mobile data.

[0788] Output: Audio data sent to the server

[0789] Step 3: Analyze the audio data

[0790] Subject: Server

[0791] Input: Audio data sent to the server

[0792] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the text generated is "Hello, um, it's a nice day today."

[0793] Output: Text data

[0794] Step 4: Detect and remove noise

[0795] Subject: Server

[0796] Input: Text data

[0797] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" and "well then" from text data. For example, the generated text is "Hello, it's a nice day today."

[0798] Output: Text data with noise removed

[0799] Step 5: Sentiment analysis and adjustment

[0800] Subject: Server

[0801] Input: Text data with noise removed

[0802] Specific operation: The server uses the emotion engine to analyze the user's emotions. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the "um" may not be removed to emphasize the angry characteristic.

[0803] Output: Text data reflecting emotions

[0804] Step 6: Generate processed audio

[0805] Subject: Server

[0806] Input: Text data reflecting emotions

[0807] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data saying "Hello, it's a nice day today" is generated.

[0808] Output: Processed audio data

[0809] Step 7: Send your processed audio

[0810] Subject: Server

[0811] Input: Processed audio data

[0812] Specific operation: The server transmits the generated processed voice to the terminal. The transmission is performed using an Internet connection, for example, using Wi-Fi or mobile data communication.

[0813] Output: Processed audio data sent to the device

[0814] Step 8: Play the processed audio

[0815] Subject: Terminal

[0816] Input: Processed voice data sent to the terminal

[0817] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear the voice saying "Hello, it's a nice day today."

[0818] Output: The processed audio played to the user

[0819] (Application example 1)

[0820] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0821] Conventional audio content delivery systems have the problem that unnecessary phrases and noises contained in the distributor's conversation are delivered as is, making the content difficult for listeners to hear. In addition, the distributor's emotions are not accurately conveyed, making it difficult to provide emotionally rich content to listeners.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0823] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for transmitting the processed voice, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed voice, and means for transmitting the generated processed voice. This makes it possible to provide listeners with audio content in real time that deletes unnecessary phrases from the distributor's conversation and emphasizes emotions.

[0824] "User conversation characteristics" refers to specific phrases or word patterns contained when a user speaks.

[0825] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[0826] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[0827] "Generative AI" refers to a system that uses artificial intelligence technology to generate text and speech.

[0828] An "emotion engine" refers to a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[0829] "Audio content distributor" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[0830] A "listener" refers to a user who receives, views, or listens to audio content.

[0831] "Real-time" refers to processing occurring simultaneously as the user's speech is being spoken.

[0832] "Tuning feature removal" refers to dynamically changing the selection of phrases or words to be removed based on the user's sentiment.

[0833] "Adjusted processed voice" refers to voice data after emotions have been enhanced and noise has been removed by the emotion engine.

[0834] A system for carrying out the present invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed voice with emphasized emotions. A specific embodiment of this system will be described below.

[0835] System Configuration

[0836] The system consists of the following main components:

[0837] 1. Speech Recognition Module: Converts the user's speech into text.

[0838] 2. Generative AI model: Analyzes the text and removes unnecessary phrases.

[0839] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[0840] 4. Speech generation module: Converts the tuned text into speech.

[0841] 5. Communication module: Transmits the generated processed audio to the listener.

[0842] Hardware and Software Used

[0843] Speech recognition module: Uses the speech_recognition library.

[0844] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[0845] Sentiment analysis engine: Uses the transformers library pipeline to leverage sentiment analysis models.

[0846] Speech generation module: Uses a Text-to-Speech (TTS) engine to convert text into speech.

[0847] Communications Module: Uses a communications protocol to transmit voice data over the Internet.

[0848] Data processing and calculation

[0849] 1. Speech recognition: Obtain user voice data and convert it to text using the speech_recognition library.

[0850] 2. Noise removal: Use a generative AI model to remove unnecessary phrases from the text.

[0851] 3. Sentiment Analysis: Using a sentiment analysis engine, we analyze user sentiment from the text.

[0852] 4. Sentiment Adjustment: Adjust the text based on the analyzed sentiment.

[0853] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[0854] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[0855] Examples

[0856] For example, if a streamer says, "Well, today I'm going to talk about my new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about my new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[0857] Examples of prompt statements

[0858] User spoken text: "Well, today I'm going to be talking about a new project. So, let me start by giving you an overview."

[0859] Sentiment Analysis Result: "Joy"

[0860] Generated text: "Today I'm going to talk about a new project. Let me start by giving you an overview."

[0861] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[0862] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0863] Step 1:

[0864] A user speaks into a microphone to deliver audio content. The terminal captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[0865] Step 2:

[0866] The device sends the captured voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data to text. The input is the voice data, and the output is the converted text data.

[0867] Step 3:

[0868] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[0869] Step 4:

[0870] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[0871] Step 5:

[0872] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, the server processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with the emotion emphasized.

[0873] Step 6:

[0874] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[0875] Step 7:

[0876] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data over the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[0877] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and voice content with emphasized emotions can be provided to the listener.

[0878] Example 2

[0879] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0880] In conventional systems, it was difficult not only to remove the characteristics of a user's conversation, but also to generate speech that accurately reflected the user's emotions. In addition, when collecting user opinions anonymously, there was a possibility that anonymity would be lost if the user's characteristic habits and phrases remained. Furthermore, it was difficult for service personnel to accurately understand the user's emotions.

[0881] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0882] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete the user's characteristic habits and phrases and collect the user's opinions while maintaining anonymity. The server also includes a means for analyzing the user's emotions and a means for generating processed voice that reflects the analyzed emotions. This makes it possible to generate a voice that accurately reflects the user's emotions, allowing the service staff to accurately understand the user's emotions.

[0883] "User conversation characteristics" refers to speech characteristics such as idiosyncrasies, phrases, and intonation that are specific to the user's speech.

[0884] "Noise" refers to distinctive habits and phrases in a user's conversations that may undermine anonymity.

[0885] "Processed voice" refers to voice data in which characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[0886] "Means of emotion analysis" refers to technologies and algorithms for detecting emotions from a user's text data or voice data and classifying and evaluating those emotions.

[0887] "Means for generating processed voice that reflects emotions" refers to technologies or algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions and generating voice data that reflects the user's emotions.

[0888] A "service representative" is a person whose role is to collect user opinions and feedback and to improve and respond to the service based on that.

[0889] "Anonymity" refers to a state in which a user's personal information and specific characteristics are protected from being known to third parties.

[0890] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0891] The present invention relates to a system for generating processed speech that reflects emotions by removing features of a user's speech. A specific embodiment of this system will be described below.

[0892] System Overview

[0893] The system receives the user's voice data and converts it into text data using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data and uses an emotion engine to analyze the user's emotions. Finally, it uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[0894] Hardware and Software Used

[0895] Speech recognition software: Google Cloud Speech-to-Text API

[0896] Natural Language Processing Libraries: spaCy, NLTK

[0897] Emotion engine: IBM Watson Tone Analyzer

[0898] Text-to-speech software: Amazon Polly

[0899] Examples

[0900] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." This system works as follows.

[0901] 1. User voice input: The user speaks into the microphone on their smartphone, saying, "Well, you see, I don't think the quality of product A is good."

[0902] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[0903] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as, "Well, then, I don't think the quality of Product A is good."

[0904] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Um, so..." from the text data, generating text data such as "I don't think the quality of Product A is good."

[0905] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[0906] 6. Generate processed voice: The server uses Amazon Polly to convert the text data, “I think the quality of product A is poor” into a processed voice with a tone that reflects sadness.

[0907] 7. Sending the processed voice: The server sends the generated processed voice to the service representative. The service representative can listen to the user's opinion anonymously through the processed voice.

[0908] Examples of prompt statements

[0909] Here is an example of a prompt to be input to the generative AI model:

[0910] User says: "Well, then, I don't think product A is of good quality."

[0911] Prompt for generative AI model: "Remove idiosyncrasies from user's speech and generate a processed voice that reflects the user's emotions."

[0912] In this way, the service representative can listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from damaging the anonymity. In addition, the processed voice reflecting the user's emotions makes it possible to accurately understand the user's emotions.

[0913] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0914] Step 1:

[0915] The user inputs speech.

[0916] A user inputs voice using a terminal (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the terminal in a digital format (e.g., WAV or MP3).

[0917] Step 2:

[0918] The server receives the audio data.

[0919] The server receives the voice data sent from the terminal. The received voice data is stored in the storage of the server. The input is the voice data from the terminal, and the output is the voice data stored in the server.

[0920] Step 3:

[0921] The server converts the voice data into text data.

[0922] The server uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[0923] Step 4:

[0924] The server removes distinctive habits and phrases from the text data.

[0925] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data from which the characteristic habits and phrases have been removed. Specifically, unnecessary phrases such as "Well, then," are deleted, and text data such as "I don't think the quality of product A is good" is generated.

[0926] Step 5:

[0927] The server analyzes the sentiment of the text data.

[0928] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from the text data. The input is text data from which characteristic habits and phrases have been removed, and the output is the analyzed emotion data. Specifically, emotions such as "sad" are analyzed from the text data.

[0929] Step 6:

[0930] The server generates processed voice that reflects the emotion.

[0931] The server uses speech synthesis software (e.g., Amazon Polly) to generate processed speech that reflects emotions. The input is the analyzed emotion data and text data, and the output is processed speech that reflects emotions. Specifically, text data such as "I think the quality of product A is not good" is converted into processed speech with a tone that reflects sadness.

[0932] Step 7:

[0933] The server transmits the processed voice to a service representative.

[0934] The server transmits the generated processed voice to the service representative. The input is the processed voice reflecting the emotion, and the output is the processed voice transmitted to the service representative. The service representative can listen to the user's opinion anonymously through the processed voice.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0937] Conventional anonymous reporting systems have problems in that a user's characteristic habits or phrases can undermine anonymity, and it is difficult to accurately convey the user's feelings. This can lead to the credibility and urgency of the report not being properly evaluated.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0939] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating a processed voice that does not include the deleted features, means for detecting the user's emotions and adjusting the processed voice based on the emotions, and means for transmitting the adjusted processed voice, thereby enabling anonymous reporting that accurately conveys emotions while maintaining the anonymity of the user.

[0940] "Characteristics of a user's conversation" refers to characteristics and habits of the user's voice when speaking, specific phrases, and the like.

[0941] "Removing as noise" refers to the process of removing characteristics of a user's speech from the voice data.

[0942] "Processed voice" refers to voice data that is generated after removing features of the user's speech.

[0943] "Means for transmitting" refers to a function for transmitting the generated processed voice to another device or server.

[0944] "Detecting emotions" refers to the process of analyzing emotions from the content of a user's speech and the tone of their voice.

[0945] "Adjusting the processed voice based on emotion" refers to the process of processing voice data to reflect detected emotion.

[0946] "Making an anonymous report" refers to the act of sending the report content while concealing the user's personal information and characteristics.

[0947] A system for implementing this invention detects characteristics of a user's speech, deletes the characteristics as noise, generates a processed voice that does not include the deleted characteristics, detects the user's emotions and adjusts the processed voice based on the emotions, and finally transmits the adjusted processed voice.

[0948] Hardware and Software Used

[0949] Hardware: Smartphone

[0950] Software: Python, speech_recognition library, pydub library, librosa library, soundfile library

[0951] Processing flow

[0952] 1. Speech recognition: The smartphone microphone is used to record the user's voice, and the speech_recognition library is used to extract text from the voice data, thereby obtaining the user's spoken content as text data.

[0953] 2. Voice processing: Using the librosa library, we remove the user's characteristic habits and phrases from the voice data. Specifically, we analyze the characteristics of the voice and remove the characteristic parts as noise.

[0954] 3. Emotion detection: Using the librosa library, we analyze the tone and pitch of the voice data to detect the user's emotions. For example, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0955] 4. Emotion-based speech adjustment: Reprocessing the speech data based on the detected emotion, for example slowing down the speech to emphasize the emotion of sadness or raising the pitch of the speech to emphasize the emotion of surprise.

[0956] 5. Audio transmission: The processed audio data is saved using the soundfile library and sent to the reporting center using the smartphone's communication function.

[0957] Examples

[0958] When a user reports, "I saw a suspicious person in my neighborhood," the system generates a processed voice saying, "I saw a suspicious person," and sends it to the reporting center. Furthermore, if the user is feeling anxious, the system can emphasize the feeling of anxiety.

[0959] Examples of prompt statements

[0960] When a user reports "I saw a suspicious person in my neighborhood," the application generates a processed voice saying "I saw a suspicious person" and sends it to the reporting center. Furthermore, if the user is feeling anxious, the application can emphasize the emotion of anxiety.

[0961] In this way, anonymous reporting that accurately conveys feelings is possible while maintaining the anonymity of the user.

[0962] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0963] Step 1:

[0964] A user records audio using the smartphone microphone. The input is the user's speech and the output is an audio file (e.g., input.wav). This audio file is used in subsequent processing steps.

[0965] Step 2:

[0966] The device uses the speech_recognition library to extract text from an audio file. The input is an audio file, and the output is text data (e.g., "I saw a suspicious person in my neighborhood."). This text data is used to obtain the user's spoken content as text information.

[0967] Step 3:

[0968] The device uses the librosa library to remove the user's characteristic habits and phrases from the voice data. The input is an audio file, and the output is the voice data with the characteristics removed. Specifically, the characteristics of the voice are analyzed, and the characteristic parts are removed as noise.

[0969] Step 4:

[0970] The device uses the librosa library to analyze the tone and pitch of the voice data to detect the user's emotions. The input is an audio file, and the output is the detected emotion (e.g., "anxiety"). Specifically, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[0971] Step 5:

[0972] The device reprocesses the speech data based on the detected emotion: the input is the feature-removed speech data and the detected emotion, and the output is processed speech with the emotion emphasized, for example slowing the speech to emphasize the emotion of sadness or raising the pitch of the speech to emphasize the emotion of surprise.

[0973] Step 6:

[0974] The device uses the soundfile library to save the processed voice data. The input is the processed voice with emphasis on emotions, and the output is the saved audio file (e.g. output.wav). This audio file is used to send to the reporting center.

[0975] Step 7:

[0976] The device uses the smartphone's communication function to send the processed voice file to the reporting center. The input is the saved voice file, and the output is the voice data sent to the reporting center. This allows anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[0977] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0978] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0979] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0980] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0981] [Third embodiment]

[0982] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0983] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0984] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0985] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0986] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0987] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0988] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0989] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0990] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0991] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0992] In the headset type terminal 314, the reception and output process is performed by the processor 46. A reception and output program 60 is stored in the storage 50. The processor 46 reads out the reception and output program 60 from the storage 50, and executes the read reception and output program 60 on the RAM 48.

[0993] The reception and output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception and output program 60 executed on the RAM 48.

[0994] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0995] "Example 1"

[0996] The present invention is a system that uses a generation AI to analyze a user's conversation in real time. This system detects features of the user's conversation, such as phrases such as "umm" and "well then," and deletes them as noise. The deletion is performed the moment the generation AI detects a characteristic phrase, and as a result, a processed voice that does not contain the features is generated. The generated processed voice is transmitted to the other party through the system.

[0997] "Example 2"

[0998] As a specific example of use, there is a case where a service representative wants to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the characteristics of the user's conversation. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask the user's opinion and prevents the user's characteristic habits and phrases from compromising the anonymity.

[0999] The process flow of each embodiment will be described below.

[1000] "Example 1"

[1001] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[1002] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[1003] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[1004] Step 4: A processed voice that does not contain the features is generated. This processed voice is sent to the other party through the system.

[1005] "Example 2"

[1006] Step 1: When a service representative wants to anonymously ask a user for feedback, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[1007] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[1008] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[1009] Step 4: A processed voice that does not include the characteristics is generated and sent to the service representative, allowing the service representative to listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from compromising the anonymity.

[1010] Example 1

[1011] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1012] Conventional voice communication systems had the problem that characteristic phrases such as "umm" and "well then" included in a user's conversation would remain as noise, making the voice difficult for the other person to hear. In addition, manually removing this noise required a lot of effort, making it impractical for real-time conversations. Furthermore, when collecting user opinions anonymously, it was difficult to accurately grasp their opinions due to the inclusion of noise.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1014] In this invention, the server includes a means for detecting features of a user's speech, a means for deleting the features as noise, and a means for generating processed speech that does not include the deleted features. This makes it possible to delete noise from a user's speech in real time and generate clear speech.

[1015] "User" refers to an individual or entity that uses the system to engage in conversation.

[1016] "Conversational features" refer to specific phrases or words such as "umm" or "well then" that are included in the user's speech.

[1017] "Noise" refers to unnecessary phrases and words that are detected as characteristic of a conversation and are targeted for removal.

[1018] "Processed audio" refers to clear audio data after noise has been removed.

[1019] "Server" refers to a computer system for processing and analyzing voice data.

[1020] "Terminal" refers to a device used by a user that records and plays audio.

[1021] "Speech recognition software" refers to a program for converting voice data into text data.

[1022] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[1023] "Text data" refers to textual information converted by voice recognition software.

[1024] "Clean text data" refers to text data after noise has been removed.

[1025] "Voice data" refers to a digital representation of a user's speech.

[1026] "Real-time" refers to processing occurring at the same time that a user conversation is taking place.

[1027] The present invention relates to a system for analyzing a user's conversation in real time and removing noise. A specific embodiment of this system will be described below.

[1028] System Configuration

[1029] Hardware

[1030] Server: Uses computer systems equipped with high-performance CPUs and GPUs, which allow for large amounts of data processing and running generative AI models.

[1031] Terminal: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[1032] software

[1033] Speech recognition software: Use common speech recognition APIs (e.g. Google Speech-to-Text API) to convert voice data into text data.

[1034] Generative AI models: Use generative AI models (e.g., OpenAI's GPT-3) to analyze user conversations and detect signature phrases.

[1035] System Operation

[1036] Acquiring and sending audio data

[1037] When a user starts a conversation, the device will record the user's voice using the built-in microphone, and the recorded voice data will be sent to the server in real time.

[1038] Converting audio data to text

[1039] The server uses voice recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted into text data as is.

[1040] Analysis and noise removal of text data

[1041] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then" are detected. These phrases are removed as noise, and clean text data is generated.

[1042] Speech conversion of clean text data

[1043] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[1044] Sending and playing processed audio

[1045] The generated processed voice is transmitted from the server to the terminal, which then plays back the received processed voice, allowing the user to hear clear voice with the noise removed.

[1046] Examples of concrete examples and prompts

[1047] For example, assume that a user has the following conversation:

[1048] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1049] The server receives this conversation and converts it into the following text data using voice recognition software:

[1050] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1051] Next, we use a generative AI model to remove noise as follows:

[1052] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[1053] Finally, the server reconverts the cleaned text data into voice data, generates processed voice, and sends it to the terminal, which plays back the processed voice.

[1054] An example of a prompt sentence might be:

[1055] "Detect characteristic phrases in user conversations and remove them as noise, such as 'um' or 'well then.'"

[1056] This prompt is then fed into a generative AI model, which then processes the user's conversation to remove noise.

[1057] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1058] Step 1:

[1059] A user initiates a conversation.

[1060] The user starts speaking to the system, for example saying, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[1061] Step 2:

[1062] The device records the user's voice and transmits it to the server.

[1063] The terminal records the user's voice using a built-in microphone. The recorded voice data is transmitted to the server in real time. The input is the user's voice data, and the output is the voice data transmitted to the server.

[1064] Step 3:

[1065] The server uses voice recognition software to convert the voice data into text data.

[1066] The server uses voice recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[1067] Step 4:

[1068] The server uses a generative AI model to analyze the text data and detect characteristic phrases.

[1069] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "well then" are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[1070] Step 5:

[1071] The server removes the detected phrases as noise and generates clean text data.

[1072] The server removes the detected characteristic phrases as noise. For example, from the text data "Well, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Well" and "Well," and generates clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[1073] Step 6:

[1074] The server reconverts the clean text data into audio data.

[1075] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk in the afternoon" is generated. The input is the cleaned text data, and the output is speech data.

[1076] Step 7:

[1077] The server transmits the generated processed voice to the terminal.

[1078] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[1079] Step 8:

[1080] The device plays the processed audio.

[1081] The terminal plays the processed voice received, and the user can hear clear voice with the noise removed. The input is the voice data sent to the terminal, and the output is the voice played from the terminal speaker.

[1082] (Application example 1)

[1083] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1084] In conventional voice distribution systems, noises such as "umm" and "well then" that occur during the streamer's conversation are transmitted to the listener, degrading the quality of the voice. In addition, streamers are required to consciously speak to avoid noise, which makes it difficult to have a natural conversation.

[1085] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete noise from conversation in real time during live streaming or voice distribution. Furthermore, by including a means for generating new text based on text from which noise has been deleted using a generative AI model, it is possible for the distributor to provide clear voice to listeners while continuing a natural conversation without consciously avoiding noise.

[1086] "User speech characteristics" refers to the particular phrases and word patterns that a user uses when speaking.

[1087] "Noise" refers to phrases or words that are unnecessary or unwanted in a conversation.

[1088] "Deletion" refers to the process of removing detected noise.

[1089] "Processed audio" refers to the clear audio after noise has been removed.

[1090] "Sending" refers to the process of delivering the generated processed voice to the other party.

[1091] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[1092] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[1093] "Generative AI model" refers to algorithms or systems that use artificial intelligence to generate text or speech.

[1094] "New text" refers to the text generated by the generative AI model after noise has been removed.

[1095] "Listener" refers to a person who receives and listens to the distributed audio.

[1096] The system for carrying out the present invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. A specific embodiment of the system will be described below.

[1097] System Configuration

[1098] This system uses the following hardware and software:

[1099] Hardware: Microphone, Smartphone

[1100] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (GPT-3 model of Hugging Face)

[1101] Processing flow

[1102] 1. Speech recognition: The voice spoken by the user through the microphone is converted into text using the SpeechRecognition library, and the characteristics of the user's speech are obtained as text data.

[1103] 2. Noise Removal: Remove noise phrases such as "um" and "well then" from the captured text data. This process is done by detecting certain phrases and removing them from the text.

[1104] 3. Generative AI model: Based on the text data with noise removed, new text is generated using a generative AI model (GPT-3). This new text has clear content with noise removed.

[1105] 4. Generate processed speech: The new text is converted to audio and generated as processed speech, which is a clearer speech with noise removed.

[1106] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive a clear audio.

[1107] Examples

[1108] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide it to listeners as "Today I'm going to talk about my new project."

[1109] Examples of prompt statements

[1110] Below are examples of prompts to be input to the generative AI model.

[1111] Remove noises like "umm" and "well then" from the user's conversation to produce clearer voice.

[1112] By inputting this prompt sentence into a generative AI model, clear voice with noise removed can be generated.

[1113] In this way, the invention removes conversation noise in real time during live streaming or audio broadcasting, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[1114] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1115] Step 1:

[1116] The device collects the voice of the user speaking through a microphone. The device converts the collected voice data into text data using the SpeechRecognition library. Through this process, the voice data is obtained as text data. The input is voice data, and the output is text data.

[1117] Step 2:

[1118] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes matching phrases. This process results in clean text data with noise removed. The input is text data containing noise, and the output is text data with noise removed.

[1119] Step 3:

[1120] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, the device uses a prompt sentence to give instructions to the generative AI model, which generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new generated text data.

[1121] Step 4:

[1122] The terminal inputs the generated new text data to a voice synthesis engine to generate processed voice. Specifically, the text data is passed to the voice synthesis engine and output as voice data. This process results in clear voice data with noise removed. The input is the generated new text data, and the output is the processed voice data.

[1123] Step 5:

[1124] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[1125] Example 2

[1126] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1127] In conventional systems, the characteristic habits and phrases of a user's conversation remain intact, which may compromise anonymity. In addition, when collecting user opinions as text, manual conversion work is required, which is inefficient. This means that it is difficult to completely protect the user's privacy when a service representative anonymously asks employees for user opinions.

[1128] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating processed voice that does not include the deleted features, a means for converting the processed voice into text, and a means for transmitting the text. This makes it possible to remove the user's characteristic habits and phrases and efficiently collect user opinions as text while maintaining anonymity.

[1129] "Characteristics of a user's conversation" refer to speech characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[1130] "Noise" refers to unnecessary features of a user's conversation that may undermine anonymity.

[1131] "Processed voice" refers to voice data that has been generated by removing noise characteristic of a user's conversation and using the remaining content.

[1132] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[1133] A "prompt" refers to an instruction entered into a generative AI model to cause it to perform a specific task.

[1134] "Means of converting to text" refers to technology or systems for analyzing voice data and converting its content into text information.

[1135] "Transmitting means" refers to technologies or systems for transmitting the generated text or voice data to other devices or systems.

[1136] A "service representative" refers to a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[1137] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[1138] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[1139] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1140] The present invention provides a system for detecting features of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[1141] Hardware and Software Used

[1142] Hardware: Servers, terminals (PCs, smartphones, etc.)

[1143] Software: Generative AI models (e.g. GPT-4), audio processing software (e.g. Audacity, Adobe Audition)

[1144] System Overview

[1145] The user uses the device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server passes the received audio file through voice processing software, which removes the user's distinctive habits and phrases. The removed voice is then generated as processed voice.

[1146] Next, the server inputs a prompt sentence to the generative AI model to convert the processed voice into text. For example, the prompt sentence is "Please convert the following voice into text: [processed voice]". The generative AI model converts the voice into text based on the prompt sentence and returns the result to the server. The server sends the converted text to the service representative. The service representative checks the received text and anonymously understands the user's opinion.

[1147] Examples

[1148] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." When the user speaks into the device's microphone, the device records this voice and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav."

[1149] The device sends the saved voice file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The server passes the received voice file to the voice processing software, which deletes unnecessary parts such as "Um, well then," and generates a processed voice that says, "I don't think the quality of Product A is good."

[1150] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates the text "I think the quality of product A is not good" and returns it to the server. The server sends this text to the service representative via email or chat app. The transmitted content is "User's opinion: I think the quality of product A is not good."

[1151] The service staff checks the received text and understands the user's opinion anonymously, which removes the user's characteristic habits and phrases.

[1152] This system allows service personnel to collect users' opinions anonymously and prevents users' characteristic habits or phrases from compromising that anonymity.

[1153] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1154] Step 1:

[1155] The user performs voice input. The user speaks into the device's microphone. For example, the user says, "Well, then, I don't think the quality of product A is good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[1156] Step 2:

[1157] The device records the voice and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's voice data, and the output is the saved digital audio file.

[1158] Step 3:

[1159] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request to the destination URL "https: / / example.com / upload". The input is the digital audio file, and the output is the audio file sent to the server.

[1160] Step 4:

[1161] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server passes the received audio file to audio processing software (e.g., Audacity's scripting function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[1162] Step 5:

[1163] The server inputs a prompt sentence to the generative AI model to convert the processed voice into text. The server inputs a prompt sentence to the generative AI model (e.g. GPT-4) saying, "Please convert the following voice into text: [processed voice]." The processed voice is attached to the prompt sentence as binary data. The input is the processed voice and the prompt sentence, and the output is the prompt sentence sent to the generative AI model.

[1164] Step 6:

[1165] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and the processed speech, and the output is the generated text.

[1166] Step 7:

[1167] The server sends the converted text to the service representative. The server sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User's opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[1168] Step 8:

[1169] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This makes it possible to obtain opinions from which the user's characteristic habits and phrases have been removed. The input is the text sent to the service representative, and the output is the confirmed user's opinion.

[1170] (Application example 2)

[1171] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1172] Conventional opinion collection systems could lose anonymity because users' characteristic habits and phrases remain intact. In addition, it was difficult to collect opinions in real time, making it difficult to collect feedback efficiently. This made it difficult for users to provide honest opinions, and there was an issue that it was difficult to obtain the information necessary to improve the service.

[1173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1174] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[1175] "User" means any person or entity that uses the System to provide input.

[1176] "Conversational features" are individually identifiable elements contained in a user's speech, such as habits, phrases, and intonation.

[1177] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[1178] "Deletion" refers to the process of removing detected noise.

[1179] "Processed voice" refers to voice data from which noise has been removed.

[1180] "Send" is the act of forwarding the processed audio to a designated recipient.

[1181] A "system" is a collection of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[1182] A "smartphone" is a portable information terminal that, in addition to having the functions of a mobile phone, can also connect to the Internet and run applications.

[1183] An "application" is a software program that runs on a smartphone.

[1184] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[1185] "Real-time" refers to data processing and analysis being carried out immediately.

[1186] "Security service" is a service aimed at protecting information and ensuring anonymity.

[1187] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[1188] The system for implementing this invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. Specifically, this system is realized using an application installed on a smartphone.

[1189] Hardware and Software Used

[1190] Hardware: Smartphone

[1191] Software: Python, speech_recognition library, gTTS library

[1192] System Operation

[1193] 1. Speech Recognition: An application installed on a smartphone collects the user's voice. The collected voice is converted to text using the speech_recognition library. In this process, a generative AI model is used to analyze the user's speech in real time.

[1194] 2. Text processing: The converted text is stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "well then," etc.).

[1195] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, a processed voice that removes the user's idiosyncrasies and phrases.

[1196] 4. Transmission: The processed voice is transmitted to a security service officer, who can collect opinions while maintaining the anonymity of the user.

[1197] Examples

[1198] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[1199] Examples of prompt statements

[1200] Example of a prompt to be input to a generative AI model:

[1201] Convert your users' speech to text and remove quirks and phrases to produce an anonymous audio.

[1202] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[1203] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1204] Step 1:

[1205] A user starts an application installed on a smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[1206] Step 2:

[1207] The device converts the collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze the user's conversation in real time. Input: Collected voice data. Output: Converted text data.

[1208] Step 3:

[1209] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[1210] Step 4:

[1211] The device converts the text data from which the noise has been removed back into speech using the gTTS library. This speech is a processed speech from which the user's characteristic habits and phrases have been removed. Input: Text data from which noise has been removed. Output: Processed speech data.

[1212] Step 5:

[1213] The terminal transmits the generated processed voice data to the security service personnel. This allows the personnel to collect opinions while maintaining the anonymity of the users. Input: Processed voice data. Output: Processed voice data transmitted to the personnel.

[1214] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[1215] "Example 1"

[1216] In one embodiment of the present invention, a system incorporating an emotion engine is provided. The system detects features of a user's speech, removes the features as noise, generates processed speech that does not include the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the removal of features based on the emotions. For example, if a user is angry, the emotion engine can generate a processed speech that does not include the removed features.

[1217] If the user is perceived as angry, the emotion engine may emphasize the angry features. If the user is perceived as happy, the emotion engine may emphasize the happy features. This will more accurately convey the user's emotions.

[1218] "Example 2"

[1219] The emotion engine also adjusts the generation of the processed voice based on the user's emotion. For example, if the user is perceived as sad, the emotion engine may emphasize the sadness characteristic. If the user is perceived as surprised, the emotion engine may emphasize the surprise characteristic. This allows the user's emotion to be conveyed more accurately.

[1220] The process flow of each embodiment will be described below.

[1221] "Example 1"

[1222] Step 1: Input the user's conversation into the emotion engine.

[1223] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1224] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[1225] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[1226] Step 5: Send the generated processed audio.

[1227] "Example 2"

[1228] Step 1: Input the user's conversation into the emotion engine.

[1229] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1230] Step 3: Adjust the generation of processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[1231] Step 4: Generate processed audio based on the emotion-based adjustment results.

[1232] Step 5: Send the generated processed audio.

[1233] Example 1

[1234] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1235] Conventional voice analysis systems have had problems in that it is difficult to effectively remove noises such as "umm" and "well then" contained in user conversations, and they also have problems in that they are unable to properly reflect the user's emotions. This often results in the user's intentions and emotions not being conveyed accurately.

[1236] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating a processed voice that does not include the deleted features, a means for transmitting the processed voice, and a means for analyzing the user's emotions and adjusting the deletion of the features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively deleting noise from the user's conversation.

[1237] "User" refers to an individual or entity that uses the system to engage in conversation.

[1238] "Conversational features" refer to specific phrases or word patterns contained in a user's speech.

[1239] "Noise" refers to unnecessary phrases or words that disrupt the flow of conversation.

[1240] "Removal" refers to the process of removing detected noise from the audio data.

[1241] "Processed audio" refers to audio data after noise has been removed.

[1242] "Transmit" refers to transferring the generated processed audio to another device or system.

[1243] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[1244] "Adjusting" refers to changing the method of noise removal based on the results of sentiment analysis.

[1245] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases or words.

[1246] The present invention is a system for analyzing a user's conversation in real time, removing noise, and generating processed voice that reflects emotions. A specific embodiment of this system will be described below.

[1247] Hardware and Software Used

[1248] Hardware: Microphones, speakers, devices (smartphones, tablets, PCs)

[1249] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[1250] System Configuration

[1251] Subject: Terminal

[1252] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[1253] Subject: Server

[1254] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects noise such as "um" and "well then" from the text data and deletes them. After deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of deleting the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, "um" may not be deleted in order to emphasize the anger characteristic.

[1255] The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology, and the generated processed voice is sent from the server to the terminal.

[1256] Subject: Terminal

[1257] The terminal reproduces the received processed voice through a speaker to the user, so that the user can hear the voice with noise removed and emotion reflected.

[1258] Examples

[1259] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. If the emotion engine analyzes the user's emotions and determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying "Hello, it's a nice day today" is generated and sent to the device. The device plays this processed voice to the user.

[1260] Examples of prompt statements

[1261] "Detect characteristic phrases such as 'um' and 'well then' from the user's speech and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, highlight the angry feature, and if the user is happy, highlight the happy feature."

[1262] The above is a specific embodiment for carrying out the present invention.

[1263] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1264] Step 1: Collect audio data

[1265] Subject: Terminal

[1266] Input: User conversation

[1267] Specific operation: When the user starts speaking, the microphone of the terminal automatically collects the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data is recorded on the terminal.

[1268] Output: Collected audio data

[1269] Step 2: Sending audio data

[1270] Subject: Terminal

[1271] Input: Collected audio data

[1272] How it works: The device transmits the collected voice data to the server in real time via an Internet connection, for example, Wi-Fi or mobile data.

[1273] Output: Audio data sent to the server

[1274] Step 3: Analyze the audio data

[1275] Subject: Server

[1276] Input: Audio data sent to the server

[1277] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the text generated is "Hello, um, it's a nice day today."

[1278] Output: Text data

[1279] Step 4: Detect and remove noise

[1280] Subject: Server

[1281] Input: Text data

[1282] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" and "well then" from text data. For example, the generated text is "Hello, it's a nice day today."

[1283] Output: Text data with noise removed

[1284] Step 5: Sentiment analysis and adjustment

[1285] Subject: Server

[1286] Input: Text data with noise removed

[1287] Specific operation: The server uses the emotion engine to analyze the user's emotions. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the "um" may not be removed to emphasize the angry characteristic.

[1288] Output: Text data reflecting emotions

[1289] Step 6: Generate processed audio

[1290] Subject: Server

[1291] Input: Text data reflecting emotions

[1292] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data saying "Hello, it's a nice day today" is generated.

[1293] Output: Processed audio data

[1294] Step 7: Send your processed audio

[1295] Subject: Server

[1296] Input: Processed audio data

[1297] Specific operation: The server transmits the generated processed voice to the terminal. The transmission is performed using an Internet connection, for example, using Wi-Fi or mobile data communication.

[1298] Output: Processed audio data sent to the device

[1299] Step 8: Play the processed audio

[1300] Subject: Terminal

[1301] Input: Processed voice data sent to the terminal

[1302] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear the voice saying "Hello, it's a nice day today."

[1303] Output: The processed audio played to the user

[1304] (Application example 1)

[1305] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1306] Conventional audio content delivery systems have the problem that unnecessary phrases and noises contained in the distributor's conversation are delivered as is, making the content difficult for listeners to hear. In addition, the distributor's emotions are not accurately conveyed, making it difficult to provide emotionally rich content to listeners.

[1307] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1308] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for transmitting the processed voice, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed voice, and means for transmitting the generated processed voice. This makes it possible to provide listeners with audio content in real time that deletes unnecessary phrases from the distributor's conversation and emphasizes emotions.

[1309] "User conversation characteristics" refers to specific phrases or word patterns contained when a user speaks.

[1310] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[1311] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[1312] "Generative AI" refers to a system that uses artificial intelligence technology to generate text and speech.

[1313] An "emotion engine" refers to a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[1314] "Audio content distributor" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[1315] A "listener" refers to a user who receives, views, or listens to audio content.

[1316] "Real-time" refers to processing occurring simultaneously as the user's speech is being spoken.

[1317] "Tuning feature removal" refers to dynamically changing the selection of phrases or words to be removed based on the user's sentiment.

[1318] "Adjusted processed voice" refers to voice data after emotions have been enhanced and noise has been removed by the emotion engine.

[1319] A system for carrying out the present invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed voice with emphasized emotions. A specific embodiment of this system will be described below.

[1320] System Configuration

[1321] The system consists of the following main components:

[1322] 1. Speech Recognition Module: Converts the user's speech into text.

[1323] 2. Generative AI model: Analyzes the text and removes unnecessary phrases.

[1324] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[1325] 4. Speech generation module: Converts the tuned text into speech.

[1326] 5. Communication module: Transmits the generated processed audio to the listener.

[1327] Hardware and Software Used

[1328] Speech recognition module: Uses the speech_recognition library.

[1329] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[1330] Sentiment analysis engine: Uses the transformers library pipeline to leverage sentiment analysis models.

[1331] Speech generation module: Uses a Text-to-Speech (TTS) engine to convert text into speech.

[1332] Communications Module: Uses a communications protocol to transmit voice data over the Internet.

[1333] Data processing and calculation

[1334] 1. Speech recognition: Obtain user voice data and convert it to text using the speech_recognition library.

[1335] 2. Noise removal: Use a generative AI model to remove unnecessary phrases from the text.

[1336] 3. Sentiment Analysis: Using a sentiment analysis engine, we analyze user sentiment from the text.

[1337] 4. Sentiment Adjustment: Adjust the text based on the analyzed sentiment.

[1338] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[1339] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[1340] Examples

[1341] For example, if a streamer says, "Well, today I'm going to talk about my new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about my new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[1342] Examples of prompt statements

[1343] User spoken text: "Well, today I'm going to be talking about a new project. So, let me start by giving you an overview."

[1344] Sentiment Analysis Result: "Joy"

[1345] Generated text: "Today I'm going to talk about a new project. Let me start by giving you an overview."

[1346] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[1347] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1348] Step 1:

[1349] A user speaks into a microphone to deliver audio content. The terminal captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[1350] Step 2:

[1351] The device sends the captured voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data to text. The input is the voice data, and the output is the converted text data.

[1352] Step 3:

[1353] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[1354] Step 4:

[1355] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[1356] Step 5:

[1357] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, the server processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with the emotion emphasized.

[1358] Step 6:

[1359] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[1360] Step 7:

[1361] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data over the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[1362] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and voice content with emphasized emotions can be provided to the listener.

[1363] Example 2

[1364] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1365] In conventional systems, it was difficult not only to remove the characteristics of a user's conversation, but also to generate speech that accurately reflected the user's emotions. In addition, when collecting user opinions anonymously, there was a possibility that anonymity would be lost if the user's characteristic habits and phrases remained. Furthermore, it was difficult for service personnel to accurately understand the user's emotions.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1367] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete the user's characteristic habits and phrases and collect the user's opinions while maintaining anonymity. The server also includes a means for analyzing the user's emotions and a means for generating processed voice that reflects the analyzed emotions. This makes it possible to generate a voice that accurately reflects the user's emotions, allowing the service staff to accurately understand the user's emotions.

[1368] "User conversation characteristics" refers to speech characteristics such as idiosyncrasies, phrases, and intonation that are specific to the user's speech.

[1369] "Noise" refers to distinctive habits and phrases in a user's conversations that may undermine anonymity.

[1370] "Processed voice" refers to voice data in which characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[1371] "Means of emotion analysis" refers to technologies and algorithms for detecting emotions from a user's text data or voice data and classifying and evaluating those emotions.

[1372] "Means for generating processed voice that reflects emotions" refers to technologies or algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions and generating voice data that reflects the user's emotions.

[1373] A "service representative" is a person whose role is to collect user opinions and feedback and to improve and respond to the service based on that.

[1374] "Anonymity" refers to a state in which a user's personal information and specific characteristics are protected from being known to third parties.

[1375] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1376] The present invention relates to a system for generating processed speech that reflects emotions by removing features of a user's speech. A specific embodiment of this system will be described below.

[1377] System Overview

[1378] The system receives the user's voice data and converts it into text data using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data and uses an emotion engine to analyze the user's emotions. Finally, it uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[1379] Hardware and Software Used

[1380] Speech recognition software: Google Cloud Speech-to-Text API

[1381] Natural Language Processing Libraries: spaCy, NLTK

[1382] Emotion engine: IBM Watson Tone Analyzer

[1383] Text-to-speech software: Amazon Polly

[1384] Examples

[1385] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." This system works as follows.

[1386] 1. User voice input: The user speaks into the microphone on their smartphone, saying, "Well, you see, I don't think the quality of product A is good."

[1387] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[1388] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as, "Well, then, I don't think the quality of Product A is good."

[1389] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Um, so..." from the text data, generating text data such as "I don't think the quality of Product A is good."

[1390] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[1391] 6. Generate processed voice: The server uses Amazon Polly to convert the text data, “I think the quality of product A is poor” into a processed voice with a tone that reflects sadness.

[1392] 7. Sending the processed voice: The server sends the generated processed voice to the service representative. The service representative can listen to the user's opinion anonymously through the processed voice.

[1393] Examples of prompt statements

[1394] Here is an example of a prompt to be input to the generative AI model:

[1395] User says: "Well, then, I don't think product A is of good quality."

[1396] Prompt for generative AI model: "Remove idiosyncrasies from user's speech and generate a processed voice that reflects the user's emotions."

[1397] In this way, the service representative can listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from damaging the anonymity. In addition, the processed voice reflecting the user's emotions makes it possible to accurately understand the user's emotions.

[1398] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1399] Step 1:

[1400] The user inputs speech.

[1401] A user inputs voice using a terminal (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the terminal in a digital format (e.g., WAV or MP3).

[1402] Step 2:

[1403] The server receives the audio data.

[1404] The server receives the voice data sent from the terminal. The received voice data is stored in the storage of the server. The input is the voice data from the terminal, and the output is the voice data stored in the server.

[1405] Step 3:

[1406] The server converts the voice data into text data.

[1407] The server uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[1408] Step 4:

[1409] The server removes distinctive habits and phrases from the text data.

[1410] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data from which the characteristic habits and phrases have been removed. Specifically, unnecessary phrases such as "Well, then," are deleted, and text data such as "I don't think the quality of product A is good" is generated.

[1411] Step 5:

[1412] The server analyzes the sentiment of the text data.

[1413] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotions from the text data. The input is text data from which characteristic habits and phrases have been removed, and the output is the analyzed emotion data. Specifically, emotions such as "sad" are analyzed from the text data.

[1414] Step 6:

[1415] The server generates processed voice that reflects the emotion.

[1416] The server uses speech synthesis software (e.g., Amazon Polly) to generate processed speech that reflects emotions. The input is the analyzed emotion data and text data, and the output is processed speech that reflects emotions. Specifically, text data such as "I think the quality of product A is not good" is converted into processed speech with a tone that reflects sadness.

[1417] Step 7:

[1418] The server transmits the processed voice to a service representative.

[1419] The server transmits the generated processed voice to the service representative. The input is the processed voice reflecting the emotion, and the output is the processed voice transmitted to the service representative. The service representative can listen to the user's opinion anonymously through the processed voice.

[1420] (Application example 2)

[1421] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1422] Conventional anonymous reporting systems have problems in that a user's characteristic habits or phrases can undermine anonymity, and it is difficult to accurately convey the user's feelings. This can lead to the credibility and urgency of the report not being properly evaluated.

[1423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1424] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating a processed voice that does not include the deleted features, means for detecting the user's emotions and adjusting the processed voice based on the emotions, and means for transmitting the adjusted processed voice, thereby enabling anonymous reporting that accurately conveys emotions while maintaining the anonymity of the user.

[1425] "Characteristics of a user's conversation" refers to characteristics and habits of the user's voice when speaking, specific phrases, and the like.

[1426] "Removing as noise" refers to the process of removing characteristics of a user's speech from the voice data.

[1427] "Processed voice" refers to voice data that is generated after removing features of the user's speech.

[1428] "Means for transmitting" refers to a function for transmitting the generated processed voice to another device or server.

[1429] "Detecting emotions" refers to the process of analyzing emotions from the content of a user's speech and the tone of their voice.

[1430] "Adjusting the processed voice based on emotion" refers to the process of processing voice data to reflect detected emotion.

[1431] "Making an anonymous report" refers to the act of sending the report content while concealing the user's personal information and characteristics.

[1432] A system for implementing this invention detects characteristics of a user's speech, deletes the characteristics as noise, generates a processed voice that does not include the deleted characteristics, detects the user's emotions and adjusts the processed voice based on the emotions, and finally transmits the adjusted processed voice.

[1433] Hardware and Software Used

[1434] Hardware: Smartphone

[1435] Software: Python, speech_recognition library, pydub library, librosa library, soundfile library

[1436] Processing flow

[1437] 1. Speech recognition: The smartphone microphone is used to record the user's voice, and the speech_recognition library is used to extract text from the voice data, thereby obtaining the user's spoken content as text data.

[1438] 2. Voice processing: Using the librosa library, we remove the user's characteristic habits and phrases from the voice data. Specifically, we analyze the characteristics of the voice and remove the characteristic parts as noise.

[1439] 3. Emotion detection: Using the librosa library, we analyze the tone and pitch of the voice data to detect the user's emotions. For example, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[1440] 4. Emotion-based speech adjustment: Reprocessing the speech data based on the detected emotion, for example slowing down the speech to emphasize the emotion of sadness or raising the pitch of the speech to emphasize the emotion of surprise.

[1441] 5. Audio transmission: The processed audio data is saved using the soundfile library and sent to the reporting center using the smartphone's communication function.

[1442] Examples

[1443] When a user reports, "I saw a suspicious person in my neighborhood," the system generates a processed voice saying, "I saw a suspicious person," and sends it to the reporting center. Furthermore, if the user is feeling anxious, the system can emphasize the feeling of anxiety.

[1444] Examples of prompt statements

[1445] When a user reports "I saw a suspicious person in my neighborhood," the application generates a processed voice saying "I saw a suspicious person" and sends it to the reporting center. Furthermore, if the user is feeling anxious, the application can emphasize the emotion of anxiety.

[1446] In this way, anonymous reporting that accurately conveys feelings is possible while maintaining the anonymity of the user.

[1447] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1448] Step 1:

[1449] A user records audio using the smartphone microphone. The input is the user's speech and the output is an audio file (e.g., input.wav). This audio file is used in subsequent processing steps.

[1450] Step 2:

[1451] The device uses the speech_recognition library to extract text from an audio file. The input is an audio file, and the output is text data (e.g., "I saw a suspicious person in my neighborhood."). This text data is used to obtain the user's spoken content as text information.

[1452] Step 3:

[1453] The device uses the librosa library to remove the user's characteristic habits and phrases from the voice data. The input is an audio file, and the output is the voice data with the characteristics removed. Specifically, the characteristics of the voice are analyzed, and the characteristic parts are removed as noise.

[1454] Step 4:

[1455] The device uses the librosa library to analyze the tone and pitch of the voice data to detect the user's emotions. The input is an audio file, and the output is the detected emotion (e.g., "anxiety"). Specifically, if the voice is low and slow, it is judged as "sadness," and if the voice is high and fast, it is judged as "surprise."

[1456] Step 5:

[1457] The device reprocesses the speech data based on the detected emotion: the input is the feature-removed speech data and the detected emotion, and the output is processed speech with the emotion emphasized, for example slowing the speech to emphasize the emotion of sadness or raising the pitch of the speech to emphasize the emotion of surprise.

[1458] Step 6:

[1459] The device uses the soundfile library to save the processed voice data. The input is the processed voice with emphasis on emotions, and the output is the saved audio file (e.g. output.wav). This audio file is used to send to the reporting center.

[1460] Step 7:

[1461] The device uses the smartphone's communication function to send the processed voice file to the reporting center. The input is the saved voice file, and the output is the voice data sent to the reporting center. This allows anonymous reporting that accurately conveys emotions while maintaining the user's anonymity.

[1462] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1463] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1464] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1465] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1466] [Fourth embodiment]

[1467] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1468] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1469] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1470] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1471] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1472] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1473] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1474] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1475] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1478] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1479] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1480] "Example 1"

[1481] The present invention is a system that uses a generation AI to analyze a user's conversation in real time. This system detects features of the user's conversation, such as phrases such as "umm" and "well then," and deletes them as noise. The deletion is performed the moment the generation AI detects a characteristic phrase, and as a result, a processed voice that does not contain the features is generated. The generated processed voice is transmitted to the other party through the system.

[1482] "Example 2"

[1483] As a specific example of use, there is a case where a service representative wants to anonymously ask employees for user opinions. In this case, the user's anonymity can be maintained by deleting the characteristics of the user's conversation. For example, if a user says, "Well, then, I don't think the quality of product A is good," the system generates a processed voice saying, "I don't think the quality of product A is good," and sends it to the service representative. This allows the service representative to anonymously ask the user's opinion and prevents the user's characteristic habits and phrases from compromising the anonymity.

[1484] The process flow of each embodiment will be described below.

[1485] "Example 1"

[1486] Step 1: The user initiates a conversation, which is sent to the system through an input device such as a microphone.

[1487] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[1488] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[1489] Step 4: A processed voice that does not contain the features is generated. This processed voice is sent to the other party through the system.

[1490] "Example 2"

[1491] Step 1: When a service representative wants to anonymously ask a user for feedback, the user initiates a conversation, which is then sent to the system via an input device such as a microphone.

[1492] Step 2: The generative AI analyzes the user's conversation in real time, including detecting characteristic phrases and habits.

[1493] Step 3: The generative AI removes characteristic phrases and habits as noise. The removal is done the moment characteristic phrases and habits are detected.

[1494] Step 4: A processed voice that does not include the characteristics is generated and sent to the service representative, allowing the service representative to listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from compromising the anonymity.

[1495] Example 1

[1496] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1497] Conventional voice communication systems had the problem that characteristic phrases such as "umm" and "well then" included in a user's conversation would remain as noise, making the voice difficult for the other person to hear. In addition, manually removing this noise required a lot of effort, making it impractical for real-time conversations. Furthermore, when collecting user opinions anonymously, it was difficult to accurately grasp their opinions due to the inclusion of noise.

[1498] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1499] In this invention, the server includes a means for detecting features of a user's speech, a means for deleting the features as noise, and a means for generating processed speech that does not include the deleted features. This makes it possible to delete noise from a user's speech in real time and generate clear speech.

[1500] "User" refers to an individual or entity that uses the system to engage in conversation.

[1501] "Conversational features" refer to specific phrases or words such as "umm" or "well then" that are included in the user's speech.

[1502] "Noise" refers to unnecessary phrases and words that are detected as characteristic of a conversation and are targeted for removal.

[1503] "Processed audio" refers to clear audio data after noise has been removed.

[1504] "Server" refers to a computer system for processing and analyzing voice data.

[1505] "Terminal" refers to a device used by a user that records and plays audio.

[1506] "Speech recognition software" refers to a program for converting voice data into text data.

[1507] "Generative AI" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases.

[1508] "Text data" refers to textual information converted by voice recognition software.

[1509] "Clean text data" refers to text data after noise has been removed.

[1510] "Voice data" refers to a digital representation of a user's speech.

[1511] "Real-time" refers to processing occurring at the same time that a user conversation is taking place.

[1512] The present invention relates to a system for analyzing a user's conversation in real time and removing noise. A specific embodiment of this system will be described below.

[1513] System Configuration

[1514] Hardware

[1515] Server: Uses computer systems equipped with high-performance CPUs and GPUs, which allow for large amounts of data processing and running generative AI models.

[1516] Terminal: A device used by a user, such as a smartphone or computer equipped with a microphone and speaker.

[1517] software

[1518] Speech recognition software: Use common speech recognition APIs (e.g. Google Speech-to-Text API) to convert voice data into text data.

[1519] Generative AI models: Use generative AI models (e.g., OpenAI's GPT-3) to analyze user conversations and detect signature phrases.

[1520] System Operation

[1521] Acquiring and sending audio data

[1522] When a user starts a conversation, the device will record the user's voice using the built-in microphone, and the recorded voice data will be sent to the server in real time.

[1523] Converting audio data to text

[1524] The server uses voice recognition software to convert the received voice data into text data. For example, if a user says, "Well, the weather is nice today. Let's go for a walk this afternoon," this is converted into text data as is.

[1525] Analysis and noise removal of text data

[1526] The server analyzes the text data using a generative AI model. As a result of the analysis, characteristic phrases such as "um" and "Well then" are detected. These phrases are removed as noise, and clean text data is generated.

[1527] Speech conversion of clean text data

[1528] The server converts the clean text data back into speech data, for example, generating speech data such as "The weather is nice today. Let's go for a walk this afternoon."

[1529] Sending and playing processed audio

[1530] The generated processed voice is transmitted from the server to the terminal, which then plays back the received processed voice, allowing the user to hear clear voice with the noise removed.

[1531] Examples of concrete examples and prompts

[1532] For example, assume that a user has the following conversation:

[1533] User: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1534] The server receives this conversation and converts it into the following text data using voice recognition software:

[1535] Text data: "Well, the weather is nice today. Let's go for a walk this afternoon."

[1536] Next, we use a generative AI model to remove noise as follows:

[1537] Clean text data: "The weather is nice today. Let's go for a walk this afternoon."

[1538] Finally, the server reconverts the cleaned text data into voice data, generates processed voice, and sends it to the terminal, which plays back the processed voice.

[1539] An example of a prompt sentence might be:

[1540] "Detect characteristic phrases in user conversations and remove them as noise, such as 'um' or 'well then.'"

[1541] This prompt is then fed into a generative AI model, which then processes the user's conversation to remove noise.

[1542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1543] Step 1:

[1544] A user initiates a conversation.

[1545] The user starts speaking to the system, for example saying, "Well, the weather is nice today. Let's go for a walk this afternoon." The input is the user's voice, and the output is the voice data input to the device's microphone.

[1546] Step 2:

[1547] The device records the user's voice and transmits it to the server.

[1548] The terminal records the user's voice using a built-in microphone. The recorded voice data is transmitted to the server in real time. The input is the user's voice data, and the output is the voice data transmitted to the server.

[1549] Step 3:

[1550] The server uses voice recognition software to convert the voice data into text data.

[1551] The server uses voice recognition software (e.g., Google Speech-to-Text API) to convert the received voice data into text data. For example, the voice data is converted into text data such as "Well, the weather is nice today. Let's go for a walk this afternoon." The input is voice data, and the output is text data.

[1552] Step 4:

[1553] The server uses a generative AI model to analyze the text data and detect characteristic phrases.

[1554] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data. As a result of the analysis, characteristic phrases such as "um" and "well then" are detected. The input is the text data, and the output is a list of the detected characteristic phrases.

[1555] Step 5:

[1556] The server removes the detected phrases as noise and generates clean text data.

[1557] The server removes the detected characteristic phrases as noise. For example, from the text data "Well, the weather is nice today. So, let's go for a walk in the afternoon," it removes "Well" and "Well," and generates clean text data "The weather is nice today. Let's go for a walk in the afternoon." The input is a list of detected characteristic phrases and the original text data, and the output is clean text data.

[1558] Step 6:

[1559] The server reconverts the clean text data into audio data.

[1560] The server reconverts the cleaned text data into speech data. For example, speech data such as "The weather is nice today. Let's go for a walk in the afternoon" is generated. The input is the cleaned text data, and the output is speech data.

[1561] Step 7:

[1562] The server transmits the generated processed voice to the terminal.

[1563] The server transmits the generated processed voice to the terminal. The transmission is performed in real time. The input is the generated voice data, and the output is the voice data transmitted to the terminal.

[1564] Step 8:

[1565] The device plays the processed audio.

[1566] The terminal plays the processed voice received, and the user can hear clear voice with the noise removed. The input is the voice data sent to the terminal, and the output is the voice played from the terminal speaker.

[1567] (Application example 1)

[1568] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1569] In conventional voice distribution systems, noises such as "umm" and "well then" that occur during the streamer's conversation are transmitted to the listener, degrading the quality of the voice. In addition, streamers are required to consciously speak to avoid noise, which makes it difficult to have a natural conversation.

[1570] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete noise from conversation in real time during live streaming or voice distribution. Furthermore, by including a means for generating new text based on text from which noise has been deleted using a generative AI model, it is possible for the distributor to provide clear voice to listeners while continuing a natural conversation without consciously avoiding noise.

[1571] "User speech characteristics" refers to the particular phrases and word patterns that a user uses when speaking.

[1572] "Noise" refers to phrases or words that are unnecessary or unwanted in a conversation.

[1573] "Deletion" refers to the process of removing detected noise.

[1574] "Processed audio" refers to the clear audio after noise has been removed.

[1575] "Sending" refers to the process of delivering the generated processed voice to the other party.

[1576] "Live streaming" refers to the delivery of audio and video in real time over the Internet.

[1577] "Audio distribution" refers to the distribution of recorded audio content over the Internet.

[1578] "Generative AI model" refers to algorithms or systems that use artificial intelligence to generate text or speech.

[1579] "New text" refers to the text generated by the generative AI model after noise has been removed.

[1580] "Listener" refers to a person who receives and listens to the distributed audio.

[1581] The system for carrying out the present invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. A specific embodiment of the system will be described below.

[1582] System Configuration

[1583] This system uses the following hardware and software:

[1584] Hardware: Microphone, Smartphone

[1585] Software: Python, SpeechRecognition library, PyAudio library, Transformers library (GPT-3 model of Hugging Face)

[1586] Processing flow

[1587] 1. Speech recognition: The voice spoken by the user through the microphone is converted into text using the SpeechRecognition library, and the characteristics of the user's speech are obtained as text data.

[1588] 2. Noise Removal: Remove noise phrases such as "um" and "well then" from the captured text data. This process is done by detecting certain phrases and removing them from the text.

[1589] 3. Generative AI model: Based on the text data with noise removed, new text is generated using a generative AI model (GPT-3). This new text has clear content with noise removed.

[1590] 4. Generate processed speech: The new text is converted to audio and generated as processed speech, which is a clearer speech with noise removed.

[1591] 5. Send: The processed audio is sent to the listener in real time, allowing the listener to receive a clear audio.

[1592] Examples

[1593] For example, if a streamer says, "Um, today I'm going to talk about my new project," the system will remove the "um" and provide it to listeners as "Today I'm going to talk about my new project."

[1594] Examples of prompt statements

[1595] Below are examples of prompts to be input to the generative AI model.

[1596] Remove noises like "umm" and "well then" from the user's conversation to produce clearer voice.

[1597] By inputting this prompt sentence into a generative AI model, clear voice with noise removed can be generated.

[1598] In this way, the invention removes conversation noise in real time during live streaming or audio broadcasting, allowing the broadcaster to continue natural conversation and provide clear audio to listeners without having to consciously avoid noise.

[1599] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1600] Step 1:

[1601] The device collects the voice of the user speaking through a microphone. The device converts the collected voice data into text data using the SpeechRecognition library. Through this process, the voice data is obtained as text data. The input is voice data, and the output is text data.

[1602] Step 2:

[1603] The device detects and removes noise phrases such as "um" and "well then" from the acquired text data. Specifically, it scans the text data using a list of noise phrases and removes matching phrases. This process results in clean text data with noise removed. The input is text data containing noise, and the output is text data with noise removed.

[1604] Step 3:

[1605] The device inputs the noise-removed text data into a generative AI model (GPT-3) to generate new text. Specifically, the device uses a prompt sentence to give instructions to the generative AI model, which generates new text based on the noise-removed text. This process results in clear text data with noise removed. The input is the noise-removed text data, and the output is the new generated text data.

[1606] Step 4:

[1607] The terminal inputs the generated new text data to a voice synthesis engine to generate processed voice. Specifically, the text data is passed to the voice synthesis engine and output as voice data. This process results in clear voice data with noise removed. The input is the generated new text data, and the output is the processed voice data.

[1608] Step 5:

[1609] The server transmits the generated processed audio data to the listener in real time. Specifically, it uploads the audio data to a streaming server and makes it accessible to the listener. This process allows the listener to receive clear audio in real time. The input is the processed audio data, and the output is the audio data transmitted to the listener.

[1610] Example 2

[1611] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1612] In conventional systems, the characteristic habits and phrases of a user's conversation remain intact, which may compromise anonymity. In addition, when collecting user opinions as text, manual conversion work is required, which is inefficient. This means that it is difficult to completely protect the user's privacy when a service representative anonymously asks employees for user opinions.

[1613] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating processed voice that does not include the deleted features, a means for converting the processed voice into text, and a means for transmitting the text. This makes it possible to remove the user's characteristic habits and phrases and efficiently collect user opinions as text while maintaining anonymity.

[1614] "Characteristics of a user's conversation" refer to speech characteristics such as idiosyncrasies, phrases, intonation, and pauses that are specific to the user's speech.

[1615] "Noise" refers to unnecessary features of a user's conversation that may undermine anonymity.

[1616] "Processed voice" refers to voice data that has been generated by removing noise characteristic of a user's conversation and using the remaining content.

[1617] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate or convert speech or text.

[1618] A "prompt" refers to an instruction entered into a generative AI model to cause it to perform a specific task.

[1619] "Means of converting to text" refers to technology or systems for analyzing voice data and converting its content into text information.

[1620] "Transmitting means" refers to technologies or systems for transmitting the generated text or voice data to other devices or systems.

[1621] A "service representative" refers to a person whose role is to analyze opinions and feedback collected from users and improve and provide services.

[1622] "Employee" refers to an individual who belongs to a particular organization or company and is employed to carry out its business.

[1623] "Anonymity" refers to a state in which a specific individual cannot be identified, and is an important element in protecting privacy.

[1624] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1625] The present invention provides a system for detecting features of user conversations and collecting opinions while maintaining anonymity. A specific embodiment of this system will be described below.

[1626] Hardware and Software Used

[1627] Hardware: Servers, terminals (PCs, smartphones, etc.)

[1628] Software: Generative AI models (e.g. GPT-4), audio processing software (e.g. Audacity, Adobe Audition)

[1629] System Overview

[1630] The user uses the device to input voice. The device records this voice and saves it as a digital audio file. The saved audio file is then sent from the device to a server. The server passes the received audio file through voice processing software, which removes the user's distinctive habits and phrases. The removed voice is then generated as processed voice.

[1631] Next, the server inputs a prompt sentence to the generative AI model to convert the processed voice into text. For example, the prompt sentence is "Please convert the following voice into text: [processed voice]". The generative AI model converts the voice into text based on the prompt sentence and returns the result to the server. The server sends the converted text to the service representative. The service representative checks the received text and anonymously understands the user's opinion.

[1632] Examples

[1633] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." When the user speaks into the device's microphone, the device records this voice and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav."

[1634] The device sends the saved voice file to the server using an HTTP POST request. The destination URL is "https: / / example.com / upload". The server passes the received voice file to the voice processing software, which deletes unnecessary parts such as "Um, well then," and generates a processed voice that says, "I don't think the quality of Product A is good."

[1635] The server inputs a prompt to the generative AI model: "Please convert the following speech into text: [processed speech]." The generative AI model generates the text "I think the quality of product A is not good" and returns it to the server. The server sends this text to the service representative via email or chat app. The transmitted content is "User's opinion: I think the quality of product A is not good."

[1636] The service staff checks the received text and understands the user's opinion anonymously, which removes the user's characteristic habits and phrases.

[1637] This system allows service personnel to collect users' opinions anonymously and prevents users' characteristic habits or phrases from compromising that anonymity.

[1638] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1639] Step 1:

[1640] The user performs voice input. The user speaks into the device's microphone. For example, the user says, "Well, then, I don't think the quality of product A is good." The input is the user's voice, and the output is the voice data captured by the device's microphone.

[1641] Step 2:

[1642] The device records the voice and saves it as a digital audio file. The device records what the user says and saves it as a digital audio file in WAV or MP3 format. The file name is in a format that includes the date and time, such as "user_input_20231010.wav". The input is the user's voice data, and the output is the saved digital audio file.

[1643] Step 3:

[1644] The device sends the recorded audio file to the server. The device sends the saved audio file to the server using an HTTP POST request to the destination URL "https: / / example.com / upload". The input is the digital audio file, and the output is the audio file sent to the server.

[1645] Step 4:

[1646] The server passes the audio file to audio processing software, which removes the user's characteristic mannerisms and phrases. The server passes the received audio file to audio processing software (e.g., Audacity's scripting function). The audio processing software removes unnecessary parts such as "Um, so..." and generates a processed voice saying "I don't think the quality of product A is good." The input is the received audio file, and the output is the processed voice.

[1647] Step 5:

[1648] The server inputs a prompt sentence to the generative AI model to convert the processed voice into text. The server inputs a prompt sentence to the generative AI model (e.g. GPT-4) saying, "Please convert the following voice into text: [processed voice]." The processed voice is attached to the prompt sentence as binary data. The input is the processed voice and the prompt sentence, and the output is the prompt sentence sent to the generative AI model.

[1649] Step 6:

[1650] The generative AI model converts the speech to text and returns the result to the server. The generative AI model converts the speech to text based on the prompt sentence, generating the text "I think the quality of product A is not good." The generated text is returned to the server. The input is the prompt sentence and the processed speech, and the output is the generated text.

[1651] Step 7:

[1652] The server sends the converted text to the service representative. The server sends the text received from the generative AI model to the service representative via email or chat app. The content sent is "User's opinion: I think the quality of product A is not good." The input is the generated text, and the output is the text sent to the service representative.

[1653] Step 8:

[1654] The service representative checks the received text. The service representative checks the received text and anonymously grasps the user's opinion. This makes it possible to obtain opinions from which the user's characteristic habits and phrases have been removed. The input is the text sent to the service representative, and the output is the confirmed user's opinion.

[1655] (Application example 2)

[1656] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1657] Conventional opinion collection systems could lose anonymity because users' characteristic habits and phrases remain intact. In addition, it was difficult to collect opinions in real time, making it difficult to collect feedback efficiently. This made it difficult for users to provide honest opinions, and there was an issue that it was difficult to obtain the information necessary to improve the service.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1659] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[1660] "User" means any person or entity that uses the System to provide input.

[1661] "Conversational features" are individually identifiable elements contained in a user's speech, such as habits, phrases, and intonation.

[1662] "Noise" refers to features of a conversation that may compromise a user's anonymity.

[1663] "Deletion" refers to the process of removing detected noise.

[1664] "Processed voice" refers to voice data from which noise has been removed.

[1665] "Send" is the act of forwarding the processed audio to a designated recipient.

[1666] A "system" is a collection of devices or software that executes a series of processes to detect characteristics of a user's speech, remove them as noise, generate processed speech, and transmit it.

[1667] A "smartphone" is a portable information terminal that, in addition to having the functions of a mobile phone, can also connect to the Internet and run applications.

[1668] An "application" is a software program that runs on a smartphone.

[1669] "Generative AI" is a system that uses artificial intelligence technology to generate or analyze data.

[1670] "Real-time" refers to data processing and analysis being carried out immediately.

[1671] "Security service" is a service aimed at protecting information and ensuring anonymity.

[1672] A "person in charge" is an individual or group whose role is to use the system to collect and analyze user opinions.

[1673] The system for implementing this invention detects the characteristics of a user's speech, removes them as noise, generates processed voice, and transmits the processed voice. Specifically, this system is realized using an application installed on a smartphone.

[1674] Hardware and Software Used

[1675] Hardware: Smartphone

[1676] Software: Python, speech_recognition library, gTTS library

[1677] System Operation

[1678] 1. Speech Recognition: An application installed on a smartphone collects the user's voice. The collected voice is converted to text using the speech_recognition library. In this process, a generative AI model is used to analyze the user's speech in real time.

[1679] 2. Text processing: The converted text is stripped of the user's unique habits and phrases. This is done by detecting and removing certain noises (e.g., "um," "well then," etc.).

[1680] 3. Speech synthesis: The processed text is converted back into speech using the gTTS library, a processed voice that removes the user's idiosyncrasies and phrases.

[1681] 4. Transmission: The processed voice is transmitted to a security service officer, who can collect opinions while maintaining the anonymity of the user.

[1682] Examples

[1683] For example, if a user says, "Well, then, I don't think the quality of product A is good," the system will generate a processed voice saying, "I don't think the quality of product A is good," and send it to the person in charge.

[1684] Examples of prompt statements

[1685] Example of a prompt to be input to a generative AI model:

[1686] Convert your users' speech to text and remove quirks and phrases to produce an anonymous audio.

[1687] In this way, it becomes possible to efficiently collect opinions in real time while maintaining the anonymity of users.

[1688] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1689] Step 1:

[1690] A user starts an application installed on a smartphone and inputs their opinion by voice. The input voice data is collected through the smartphone's microphone. Input: User's voice data. Output: Collected voice data.

[1691] Step 2:

[1692] The device converts the collected voice data into text using the speech_recognition library. This process uses a generative AI model to analyze the user's conversation in real time. Input: Collected voice data. Output: Converted text data.

[1693] Step 3:

[1694] The device detects characteristic habits and phrases from the converted text data and removes them as noise. This process is performed by detecting and removing specific noise (e.g., "um," "Well then," etc.). Input: Converted text data. Output: Text data with noise removed.

[1695] Step 4:

[1696] The device converts the text data from which the noise has been removed back into speech using the gTTS library. This speech is a processed speech from which the user's characteristic habits and phrases have been removed. Input: Text data from which noise has been removed. Output: Processed speech data.

[1697] Step 5:

[1698] The terminal transmits the generated processed voice data to the security service personnel. This allows the personnel to collect opinions while maintaining the anonymity of the users. Input: Processed voice data. Output: Processed voice data transmitted to the personnel.

[1699] In addition, an emotion engine that estimates the emotion of the user may be further combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[1700] "Example 1"

[1701] As an embodiment of the present invention, a system incorporating an emotion engine is provided. The system detects features of a user's speech, removes the features as noise, generates processed speech without the removed features, and transmits the processed speech. In particular, the emotion engine analyzes emotions from the user's speech and adjusts the removal of features based on the emotions. For example, if the user is perceived as angry, the emotion engine may emphasize the angry features. Also, if the user is perceived as happy, the emotion engine may emphasize the happy features. This allows the user's emotions to be conveyed more accurately.

[1702] "Example 2"

[1703] The emotion engine also adjusts the generation of the processed voice based on the user's emotion. For example, if the user is perceived as sad, the emotion engine may emphasize the sadness characteristic. If the user is perceived as surprised, the emotion engine may emphasize the surprise characteristic. This allows the user's emotion to be conveyed more accurately.

[1704] The process flow of each embodiment will be described below.

[1705] "Example 1"

[1706] Step 1: Input the user's conversation into the emotion engine.

[1707] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1708] Step 3: Adjust feature removal based on the analyzed emotion. For example, if the user is perceived as angry, the emotion engine will highlight the angry feature.

[1709] Step 4: Generate processed speech based on the results of adjusting the feature removal.

[1710] Step 5: Send the generated processed audio.

[1711] "Example 2"

[1712] Step 1: Input the user's conversation into the emotion engine.

[1713] Step 2: The emotion engine analyzes emotions from the user's conversation.

[1714] Step 3: Adjust the generation of processed voice based on the analyzed emotion. For example, if the user feels sad, the emotion engine will emphasize the sadness characteristics.

[1715] Step 4: Generate processed audio based on the emotion-based adjustment results.

[1716] Step 5: Send the generated processed audio.

[1717] Example 1

[1718] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1719] Conventional voice analysis systems have had problems in that it is difficult to effectively remove noises such as "umm" and "well then" contained in user conversations, and they also have problems in that they are unable to properly reflect the user's emotions. This often results in the user's intentions and emotions not being conveyed accurately.

[1720] The identification process by the identification processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means. In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, a means for generating a processed voice that does not include the deleted features, a means for transmitting the processed voice, and a means for analyzing the user's emotions and adjusting the deletion of the features based on the emotions. This makes it possible to generate voice data that appropriately reflects the user's emotions while effectively deleting noise from the user's conversation.

[1721] "User" refers to an individual or entity that uses the system to engage in conversation.

[1722] "Conversational features" refer to specific phrases or word patterns contained in a user's speech.

[1723] "Noise" refers to unnecessary phrases or words that disrupt the flow of conversation.

[1724] "Removal" refers to the process of removing detected noise from the audio data.

[1725] "Processed audio" refers to audio data after noise has been removed.

[1726] "Transmit" refers to transferring the generated processed audio to another device or system.

[1727] "Sentiment analysis" refers to the process of identifying emotions from a user's speech.

[1728] "Adjusting" refers to changing the method of noise removal based on the results of sentiment analysis.

[1729] A "generative AI model" refers to an artificial intelligence model that analyzes user conversations and detects specific phrases or words.

[1730] The present invention is a system for analyzing a user's conversation in real time, removing noise, and generating processed voice that reflects emotions. A specific embodiment of this system will be described below.

[1731] Hardware and Software Used

[1732] Hardware: Microphones, speakers, devices (smartphones, tablets, PCs)

[1733] Software: Generative AI models (e.g., OpenAI's GPT-4), emotion engines (e.g., Affectiva's emotion recognition API)

[1734] System Configuration

[1735] Subject: Terminal

[1736] The device is equipped with a microphone to collect the user's speech. When the user starts speaking, the device's microphone automatically collects the voice and transmits it to the server in real time. The device temporarily stores the collected voice data in its internal memory and transmits it to the server via an Internet connection.

[1737] Subject: Server

[1738] The server analyzes the received voice data using a generative AI model. During the analysis process, the voice data is converted into text data. The generative AI model detects noise such as "um" and "well then" from the text data and deletes them. After deletion, the text data is analyzed for the user's emotions by an emotion engine. The method of deleting the noise is adjusted based on the results of the emotion analysis. For example, if the user is angry, "um" may not be deleted in order to emphasize the anger characteristic.

[1739] The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology, and the generated processed voice is sent from the server to the terminal.

[1740] Subject: Terminal

[1741] The terminal reproduces the received processed voice through a speaker to the user, so that the user can hear the voice with noise removed and emotion reflected.

[1742] Examples

[1743] For example, if a user says, "Hello, um, it's a nice day today," the voice data is recorded on the device and sent to the server. The server uses a generative AI model to detect the "um" as noise and removes it. If the emotion engine analyzes the user's emotions and determines that the user is happy, it emphasizes the characteristics of that happiness. Finally, a processed voice saying "Hello, it's a nice day today" is generated and sent to the device. The device plays this processed voice to the user.

[1744] Examples of prompt statements

[1745] "Detect characteristic phrases such as 'um' and 'well then' from the user's speech and remove them as noise. Also, analyze the user's emotions and adjust feature removal based on the emotions. For example, if the user is angry, highlight the angry feature, and if the user is happy, highlight the happy feature."

[1746] The above is a specific embodiment for carrying out the present invention.

[1747] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1748] Step 1: Collect audio data

[1749] Subject: Terminal

[1750] Input: User conversation

[1751] Specific operation: When the user starts speaking, the microphone of the terminal automatically collects the voice. For example, when the user says, "Hello, well, it's a nice day today," the voice data is recorded on the terminal.

[1752] Output: Collected audio data

[1753] Step 2: Sending audio data

[1754] Subject: Terminal

[1755] Input: Collected audio data

[1756] How it works: The device transmits the collected voice data to the server in real time via an Internet connection, for example, Wi-Fi or mobile data.

[1757] Output: Audio data sent to the server

[1758] Step 3: Analyze the audio data

[1759] Subject: Server

[1760] Input: Audio data sent to the server

[1761] Specific operation: The server analyzes the received voice data using the generative AI model. During the analysis process, the voice data is converted into text data. For example, the text generated is "Hello, um, it's a nice day today."

[1762] Output: Text data

[1763] Step 4: Detect and remove noise

[1764] Subject: Server

[1765] Input: Text data

[1766] Specific operation: The server uses a generative AI model to detect and remove noise such as "um" and "well then" from text data. For example, the generated text is "Hello, it's a nice day today."

[1767] Output: Text data with noise removed

[1768] Step 5: Sentiment analysis and adjustment

[1769] Subject: Server

[1770] Input: Text data with noise removed

[1771] Specific operation: The server uses the emotion engine to analyze the user's emotions. Based on the analysis results, the noise removal method is adjusted. For example, if the user is angry, the "um" may not be removed to emphasize the angry characteristic.

[1772] Output: Text data reflecting emotions

[1773] Step 6: Generate processed audio

[1774] Subject: Server

[1775] Input: Text data reflecting emotions

[1776] Specific operation: The server converts the noise-removed text data back into voice data using text-to-speech synthesis technology. For example, voice data saying "Hello, it's a nice day today" is generated.

[1777] Output: Processed audio data

[1778] Step 7: Send your processed audio

[1779] Subject: Server

[1780] Input: Processed audio data

[1781] Specific operation: The server transmits the generated processed voice to the terminal. The transmission is performed using an Internet connection, for example, using Wi-Fi or mobile data communication.

[1782] Output: Processed audio data sent to the device

[1783] Step 8: Play the processed audio

[1784] Subject: Terminal

[1785] Input: Processed voice data sent to the terminal

[1786] Specific operation: The terminal plays the received processed voice to the user through the speaker, for example, the user can hear the voice saying "Hello, it's a nice day today."

[1787] Output: The processed audio played to the user

[1788] (Application example 1)

[1789] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1790] Conventional audio content delivery systems have the problem that unnecessary phrases and noises contained in the distributor's conversation are delivered as is, making the content difficult for listeners to hear. In addition, the distributor's emotions are not accurately conveyed, making it difficult to provide emotionally rich content to listeners.

[1791] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1792] In this invention, the server includes means for detecting features of a user's conversation, means for deleting the features as noise, means for generating processed voice that does not include the deleted features, means for transmitting the processed voice, means for analyzing the user's emotions, means for adjusting the deletion of the features based on the emotions, means for generating the adjusted processed voice, and means for transmitting the generated processed voice. This makes it possible to provide listeners with audio content in real time that deletes unnecessary phrases from the distributor's conversation and emphasizes emotions.

[1793] "User conversation characteristics" refers to specific phrases or word patterns contained when a user speaks.

[1794] "Noise" refers to phrases or words that are unnecessary or meaningless in a user's conversation.

[1795] "Processed voice" refers to voice data after noise has been removed from the user's conversation and emotions have been emphasized as necessary.

[1796] "Generative AI" refers to a system that uses artificial intelligence technology to generate text and speech.

[1797] An "emotion engine" refers to a system that analyzes emotions from a user's conversation and adjusts voice and text based on those emotions.

[1798] "Audio content distributor" refers to a person or organization that provides audio content, such as podcasts or live streaming, to listeners.

[1799] A "listener" refers to a user who receives, views, or listens to audio content.

[1800] "Real-time" refers to processing occurring simultaneously as the user's speech is being spoken.

[1801] "Tuning feature removal" refers to dynamically changing the selection of phrases or words to be removed based on the user's sentiment.

[1802] "Adjusted processed voice" refers to voice data after emotions have been enhanced and noise has been removed by the emotion engine.

[1803] A system for carrying out the present invention analyzes a user's conversation in real time, deletes unnecessary phrases, and generates processed voice with emphasized emotions. A specific embodiment of this system will be described below.

[1804] System Configuration

[1805] The system consists of the following main components:

[1806] 1. Speech Recognition Module: Converts the user's speech into text.

[1807] 2. Generative AI model: Analyzes the text and removes unnecessary phrases.

[1808] 3. Sentiment Analysis Engine: Analyzes user sentiment and adjusts text based on sentiment.

[1809] 4. Speech generation module: Converts the tuned text into speech.

[1810] 5. Communication module: Transmits the generated processed audio to the listener.

[1811] Hardware and Software Used

[1812] Speech recognition module: Uses the speech_recognition library.

[1813] Generative AI models: Use the transformers library pipeline to utilize generative AI models such as GPT-3.

[1814] Sentiment analysis engine: Uses the transformers library pipeline to leverage sentiment analysis models.

[1815] Speech generation module: Uses a Text-to-Speech (TTS) engine to convert text into speech.

[1816] Communications Module: Uses a communications protocol to transmit voice data over the Internet.

[1817] Data processing and calculation

[1818] 1. Speech recognition: Obtain user voice data and convert it to text using the speech_recognition library.

[1819] 2. Noise removal: Use a generative AI model to remove unnecessary phrases from the text.

[1820] 3. Sentiment Analysis: Using a sentiment analysis engine, we analyze user sentiment from the text.

[1821] 4. Sentiment Adjustment: Adjust the text based on the analyzed sentiment.

[1822] 5. Speech generation: Convert the adjusted text into speech using a TTS engine.

[1823] 6. Audio transmission: The generated audio data is transmitted to the listener via the communication module.

[1824] Examples

[1825] For example, if a streamer says, "Well, today I'm going to talk about my new project. So, let me start by giving you an overview," the system will translate it to, "Today I'm going to talk about my new project. Let me start by giving you an overview," and emphasize the necessary emotions based on sentiment analysis.

[1826] Examples of prompt statements

[1827] User spoken text: "Well, today I'm going to be talking about a new project. So, let me start by giving you an overview."

[1828] Sentiment Analysis Result: "Joy"

[1829] Generated text: "Today I'm going to talk about a new project. Let me start by giving you an overview."

[1830] In this way, the broadcaster's voice can be analyzed in real time, unnecessary phrases can be removed, and emotions can be emphasized to provide more engaging content for listeners.

[1831] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1832] Step 1:

[1833] A user speaks into a microphone to deliver audio content. The terminal captures this audio data. The input is the user's audio data, and the output is the captured audio data.

[1834] Step 2:

[1835] The device sends the captured voice data to the voice recognition module, which uses the speech_recognition library to convert the voice data to text. The input is the voice data, and the output is the converted text data.

[1836] Step 3:

[1837] The server uses a generative AI model to remove unnecessary phrases from text data. The generative AI model uses a pipeline from the transformers library. The input is text data, and the output is text data with unnecessary phrases removed.

[1838] Step 4:

[1839] The server uses a sentiment analysis engine to analyze user sentiment from text data. The sentiment analysis engine also uses a pipeline from the transformers library. The input is text data with unnecessary phrases removed, and the output is the analyzed sentiment data.

[1840] Step 5:

[1841] The server adjusts the text data based on the analyzed emotion data. Using the results of the emotion analysis engine, the server processes the text data to emphasize emotions. The input is the text data with unnecessary phrases removed and emotion data, and the output is the text data with the emotion emphasized.

[1842] Step 6:

[1843] The server sends the adjusted text data to the speech generation module, which uses a Text-to-Speech (TTS) engine to convert the text data into speech. The input is the emotion-enhanced text data, and the output is the generated speech data.

[1844] Step 7:

[1845] The server transmits the generated audio data to the listener via the communication module, which transmits the audio data over the Internet. The input is the generated audio data, and the output is the audio data transmitted to the listener.

[1846] In this way, the user's voice can be analyzed in real time, unnecessary phrases can be deleted, and voice content with emphasized emotions can be provided to the listener.

[1847] Example 2

[1848] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1849] In conventional systems, it was difficult not only to remove the characteristics of a user's conversation, but also to generate speech that accurately reflected the user's emotions. In addition, when collecting user opinions anonymously, there was a possibility that anonymity would be lost if the user's characteristic habits and phrases remained. Furthermore, it was difficult for service personnel to accurately understand the user's emotions.

[1850] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1851] In this invention, the server includes a means for detecting features of a user's conversation, a means for deleting the features as noise, and a means for generating processed voice that does not include the deleted features. This makes it possible to delete the user's characteristic habits and phrases and collect the user's opinions while maintaining anonymity. The server also includes a means for analyzing the user's emotions and a means for generating processed voice that reflects the analyzed emotions. This makes it possible to generate a voice that accurately reflects the user's emotions, allowing the service staff to accurately understand the user's emotions.

[1852] "User conversation characteristics" refers to speech characteristics such as idiosyncrasies, phrases, and intonation that are specific to the user's speech.

[1853] "Noise" refers to distinctive habits and phrases in a user's conversations that may undermine anonymity.

[1854] "Processed voice" refers to voice data in which characteristics of the user's conversation have been removed and emotions have been reflected as necessary.

[1855] "Means of emotion analysis" refers to technologies and algorithms for detecting emotions from a user's text data or voice data and classifying and evaluating those emotions.

[1856] "Means for generating processed voice that reflects emotions" refers to technologies or algorithms for adjusting the tone and intonation of the voice based on the analyzed emotions and generating voice data that reflects the user's emotions.

[1857] A "service representative" is a person whose role is to collect user opinions and feedback and to improve and respond to the service based on that.

[1858] "Anonymity" refers to a state in which a user's personal information and specific characteristics are protected from being known to third parties.

[1859] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1860] The present invention relates to a system for generating processed speech that reflects emotions by removing features of a user's speech. A specific embodiment of this system will be described below.

[1861] System Overview

[1862] The system receives the user's voice data and converts it into text data using speech recognition software. It then uses natural language processing technology to remove quirks and phrases from the text data and uses an emotion engine to analyze the user's emotions. Finally, it uses speech synthesis software to generate a processed voice that reflects the user's emotions and sends it to the service representative.

[1863] Hardware and Software Used

[1864] Speech recognition software: Google Cloud Speech-to-Text API

[1865] Natural Language Processing Libraries: spaCy, NLTK

[1866] Emotion engine: IBM Watson Tone Analyzer

[1867] Text-to-speech software: Amazon Polly

[1868] Examples

[1869] Consider a case where a user says, "Well, then, I don't think the quality of product A is good." This system works as follows.

[1870] 1. User voice input: The user speaks into the microphone on their smartphone, saying, "Well, you see, I don't think the quality of product A is good."

[1871] 2. Receiving audio data: The server receives the audio data (e.g., WAV format) sent from the smartphone.

[1872] 3. Speech recognition: The server uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as, "Well, then, I don't think the quality of Product A is good."

[1873] 4. Removal of characteristic quirks: The server uses spaCy to remove unnecessary phrases such as "Um, so..." from the text data, generating text data such as "I don't think the quality of Product A is good."

[1874] 5. Sentiment analysis: The server uses IBM Watson Tone Analyzer to analyze the user's emotion from the text data and determines that the emotion is "sad."

[1875] 6. Generate processed voice: The server uses Amazon Polly to convert the text data, “I think the quality of product A is poor” into a processed voice with a tone that reflects sadness.

[1876] 7. Sending the processed voice: The server sends the generated processed voice to the service representative. The service representative can listen to the user's opinion anonymously through the processed voice.

[1877] Examples of prompt statements

[1878] Here is an example of a prompt to be input to the generative AI model:

[1879] User says: "Well, then, I don't think product A is of good quality."

[1880] Prompt for generative AI model: "Remove idiosyncrasies from user's speech and generate a processed voice that reflects the user's emotions."

[1881] In this way, the service representative can listen to the user's opinion anonymously and prevent the user's characteristic habits and phrases from damaging the anonymity. In addition, the processed voice reflecting the user's emotions makes it possible to accurately understand the user's emotions.

[1882] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1883] Step 1:

[1884] The user inputs speech.

[1885] A user inputs voice using a terminal (e.g., a smartphone or a PC). Voice input is performed through a microphone. The input voice data is stored in the terminal in a digital format (e.g., WAV or MP3).

[1886] Step 2:

[1887] The server receives the audio data.

[1888] The server receives the voice data sent from the terminal. The received voice data is stored in the storage of the server. The input is the voice data from the terminal, and the output is the voice data stored in the server.

[1889] Step 3:

[1890] The server converts the voice data into text data.

[1891] The server uses voice recognition software (e.g., Google Cloud Speech-to-Text API) to convert the received voice data into text data. The input is voice data, and the output is text data. Specifically, the voice data is converted into text data such as "Well, then, I think the quality of product A is not good."

[1892] Step 4:

[1893] The server removes distinctive habits and phrases from the text data.

[1894] The server uses natural language processing (NLP) technology to remove the user's characteristic habits and phrases from the text data. For this process, libraries such as spaCy and NLTK are used. The input is text data, and the output is text data from which the characteristic habits and phrases have been removed. Specifically, unnecessary phrases such as "Well, then," are deleted, and text data such as "I don't think the quality of product A is good" is generated.

[1895] Step 5:

[1896] The server analyzes the sentiment of the text data.

[1897] The server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze ...

Claims

1. A means for detecting characteristics of a user's conversation using a generative AI model; A deletion means for deleting the features included in the conversation as noise by inputting a specific prompt sentence into a generative AI model; A means for generating a processed voice that does not include the features deleted as noise as a clear voice that does not include the noise; means for transmitting the processed audio to a listener who hears the conversation; an emotion engine that recognizes the emotion of the user; means for dynamically changing a selection of phrases and words to be deleted from the conversation based on the emotion; Including, the system.

2. The system according to claim 1 , wherein the deletion means deletes the features included in the conversation during live streaming over the Internet as noise.

3. The system of claim 1 , wherein the noise is a characteristic quirk or phrase in the conversation that may compromise the user's anonymity.

Images