System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving audio data; means for denoising and loudness normalizing the audio data; means for converting the audio data into text data; means for performing sentiment analysis on the text data; means for evaluating performance based on results of the sentiment analysis and keyword extraction; and means for generating specific feedback based on evaluation results and providing the feedback to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Improving the performance of sales representatives and call center staff during business calls and sales conversations is not easy, especially since it is difficult to visualize specific issues with the quality of conversations and communication using data. As a result, it is not clear what the representative needs to improve, making it difficult to implement effective data-driven coaching. Furthermore, traditional methods for improving communication skills are uniform and do not allow for optimization based on individual skills and characteristics. For this reason, companies need to equip each representative with the optimal conversation skills to improve business performance. [Means for solving the problem]

[0005] This invention proposes a system for visualizing the differences in behavior and speech between high and low performers based on information extracted from voice data. The system primarily includes means for receiving voice data, removing noise and normalizing the volume, and converting the voice data into text. Furthermore, the system performs sentiment analysis on the text data and evaluates performance based on the results of the sentiment analysis and keyword extraction. Finally, the system includes means for generating and providing specific feedback to users based on the evaluation results, thereby providing specific advice tailored to each employee's individual skills and continuously optimizing their communication skills.

[0006] "Audio data" refers to digital audio files that contain recorded conversations such as business calls and sales talks made by users.

[0007] "Noise reduction" is a process that removes background noise and unnecessary sounds from audio data to improve audio quality.

[0008] "Volume normalization" is a process that equalizes the overall volume level of audio data to make it easier to hear.

[0009] "Text data" refers to character information converted from voice data using voice recognition technology.

[0010] "Sentiment analysis" is a process that uses natural language processing technology to analyze the emotional tone (positive, negative, neutral, etc.) of a speaker on text data.

[0011] "Keyword extraction" is a process that clarifies the main points and focus of a conversation by picking out important words and phrases in the conversation.

[0012] "Performance evaluation" is the process of quantifying and evaluating the effectiveness of a user's business calls or sales pitches based on text data and voice characteristics.

[0013] "Feedback" refers to specific advice and areas for improvement to improve the user's communication skills based on the results of the performance evaluation.

[0014] A "high performer" is someone who demonstrates high performance in business calls and sales talks.

[0015] A "low performer" is a person who exhibits poor performance in business calls and sales talks. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that receives voice data and analyzes it to improve a user's communication skills. The system program will be explained below, and its processing will be described with specific examples.

[0038] System Overview

[0039] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, evaluates the performance, and then provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[0040] Program processing overview

[0041] server:

[0042] 1. Receiving audio data:

[0043] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[0044] 2. Preprocessing the audio data:

[0045] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[0046] 3. Transcription of audio data:

[0047] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[0048] 4. Sentiment analysis and keyword extraction:

[0049] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[0050] 5. Performance Evaluation:

[0051] The server scores users' performance based on sentiment analysis and keyword extraction. The AI ​​model learns from past data and identifies patterns of high and low performers.

[0052] 6. Feedback Generation:

[0053] The server generates specific feedback based on the results of the performance evaluation, including areas for improvement and specific advice.

[0054] 7. Submitting Feedback:

[0055] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[0056] Device:

[0057] 1. Audio recording and transmission:

[0058] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[0059] 2. Receiving Feedback:

[0060] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[0061] User:

[0062] 1. Start recording:

[0063] The user launches the application on the terminal and starts recording business calls or sales talks.

[0064] 2. Feedback check and implementation:

[0065] Users review the feedback they receive and implement improvements in their next call or conversation.

[0066] Specific examples

[0067] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[0068] 1. Audio Recording:

[0069] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[0070] 2. Sending audio data:

[0071] After the conversation ends, the device automatically sends the recording data to the server.

[0072] 3. Data analysis and feedback generation:

[0073] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data, performs sentiment analysis and keyword extraction, extracts points that need emphasis during the conversation, and generates feedback.

[0074] 4. Feedback received:

[0075] The device receives feedback from the server and provides specific advice such as "ask more open-ended questions to further draw out the customer's concerns."

[0076] 5. Implement improvements:

[0077] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[0078] In this way, users can continuously optimize their communication skills through data-driven coaching using AI. The system of the present invention can also significantly contribute to improving a company's business performance.

[0079] The processing flow will be explained below.

[0080] Step 1:

[0081] User:

[0082] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[0083] Step 2:

[0084] Device:

[0085] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[0086] Step 3:

[0087] Device:

[0088] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[0089] Step 4:

[0090] server:

[0091] The server receives the voice data transmitted from the terminal.

[0092] Step 5:

[0093] server:

[0094] The server performs noise reduction and volume normalization on the received audio data, improving the sound quality of the recorded data.

[0095] Step 6:

[0096] server:

[0097] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[0098] Step 7:

[0099] server:

[0100] The server uses natural language processing technology to perform sentiment analysis on the generated text data, classifying the emotional tone of the utterance as positive, negative, neutral, etc.

[0101] Step 8:

[0102] server:

[0103] The server simultaneously extracts keywords and important phrases from the text data, making the main points of the conversation clearer.

[0104] Step 9:

[0105] server:

[0106] The server evaluates user performance based on text data, sentiment analysis, and keyword extraction. Using an AI model trained on past data, it scores users based on the patterns of high and low performers.

[0107] Step 10:

[0108] server:

[0109] Based on the results of the performance evaluation, the server generates feedback to the user, including areas for improvement and specific advice.

[0110] Step 11:

[0111] server:

[0112] The server transmits the generated feedback to the user's terminal.

[0113] Step 12:

[0114] Device:

[0115] The terminal receives feedback from the server and visualizes it for the user to understand.

[0116] Step 13:

[0117] User:

[0118] The user can check the feedback displayed on the device and use it to make improvements in their next business call or sales conversation.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] Conventional systems for improving communication skills that use voice data have the problem of being unable to consistently perform advanced processing such as noise removal, volume normalization, emotion analysis, and keyword extraction when analyzing user voice data. Furthermore, the inability to provide real-time evaluation or feedback makes it difficult to quickly improve users' communication skills.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis and keyword extraction on the text data, means for evaluating performance based on the results of the sentiment analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for visualizing the feedback through a user interface. This makes it possible to perform advanced voice data preprocessing and natural language processing in real time and provide specific feedback to the user.

[0124] "Audio data" refers to data in which audio is recorded in digital format.

[0125] A "receiving means" is a device or program that has the function of receiving specific data from another device or system.

[0126] "Noise reduction" is the process of reducing or eliminating unwanted background noise in audio data.

[0127] "Volume normalization" is a process for making the volume level of audio data uniform.

[0128] "Means for converting to text data" refers to a system or algorithm for converting audio data into a corresponding string of characters.

[0129] "Sentiment analysis" is the process of analyzing and extracting the type and intensity of emotions from text data.

[0130] "Keyword extraction" is a technique for extracting important words and phrases from text data.

[0131] A "means for evaluating performance" is a device or program that has the function of evaluating a user's performance or results based on collected data.

[0132] "Feedback" refers to specific advice or suggestions provided to the user based on the evaluation results.

[0133] A "user interface" is the visual or operational means by which a user interacts with a system.

[0134] The present invention is a system that receives and analyzes voice data to improve a user's communication skills. This system is mainly composed of a "server," a "terminal," and a "user."

[0135] Hardware and Software Overview

[0136] server:

[0137] The server plays a central role in processing the audio data. The following software and tools are used for this processing:

[0138] Audio processing library: Librosa

[0139] Speech recognition engine: Google Cloud Speech-to-Text API

[0140] Natural language processing libraries: spaCy, NLTK

[0141] Machine learning libraries: Scikit-learn, TensorFlow

[0142] Device:

[0143] The terminal is a device that allows users to record audio and send and receive data to and from the server. Typical terminals are smartphones or tablets with built-in recording applications.

[0144] Processing flow and specific examples

[0145] Audio recording and transmission

[0146] The user starts the recording application on the device and records business calls or sales talks. The audio is temporarily stored on the device, and once recording is complete, it is encrypted and sent to the server.

[0147] Receiving and preprocessing audio data

[0148] The server receives the encrypted audio data, which is then denoised and normalised using the Librosa library, improving the quality of the audio data.

[0149] Conversion to text data

[0150] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data, which is then used for sentiment analysis and keyword extraction.

[0151] Sentiment analysis and keyword extraction

[0152] The server analyzes emotions from the transcribed text data and extracts important keywords. This process uses the spaCy and NLTK libraries. For example, it determines whether the emotion contained in the text data is "joy" or "anger."

[0153] Performance evaluation and feedback generation

[0154] The server evaluates the user's performance based on the results of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn and TensorFlow models trained on past data. Specific feedback is generated based on the evaluation results. For example, advice such as "You should ask more open-ended questions to further elicit customer concerns" is provided.

[0155] Send and visualize feedback

[0156] The server transmits the generated feedback to the user's terminal, which receives the feedback, visualizes it through a user interface, and presents it to the user.

[0157] Specific use cases

[0158] For example, the following scenario shows how a sales representative might use the system when approaching a new business deal.

[0159] 1. The sales representative (user) starts recording on the device while talking to the client.

[0160] 2. When the conversation ends, the device encrypts the recording and sends it to the server.

[0161] 3. The server receives the audio data and performs preprocessing, converting the audio into text data, and performing sentiment analysis and keyword extraction.

[0162] 4. The server generates feedback such as "You should simplify the explanations in areas where the customer is not interested."

[0163] 5. The device receives this feedback and presents it to the user in a visual form.

[0164] 6. The salesperson (user) uses this feedback in the next sales meeting to improve communication.

[0165] Prompt Sentence Examples

[0166] "Please analyze the audio data of this sales pitch and extract user sentiment and keywords. Please also include areas for improvement and specific advice in your feedback."

[0167] "Perform sentiment analysis on the following text data, extract keywords, and generate feedback: 'The customer asked many questions during the product explanation, which we answered well. However, you should ask more open-ended questions.'"

[0168] The above is a specific embodiment for carrying out the present invention. This system allows users to receive feedback in real time, enabling them to quickly improve their communication skills.

[0169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0170] Step 1:

[0171] Audio recording and transmission

[0172] Device: The user launches the device's recording application and records a business call or sales conversation. Specifically, the device uses the built-in microphone to collect voice data (input) and temporarily stores it within the application. After the recording is complete, the voice data is encrypted. The encrypted voice data (output) is sent to the server using the HTTPS protocol.

[0173] Step 2:

[0174] Receiving audio data

[0175] Server: The server receives the encrypted voice data (input) sent from the device. Specifically, the server receives the data using the HTTPS protocol and stores it securely in a database. The stored voice data (output) is used for the next preprocessing step.

[0176] Step 3:

[0177] Audio data preprocessing

[0178] Server: The server performs noise reduction, volume normalization, and speaker separation on the stored audio data (input). This is done using the Librosa library. Specifically, the server analyzes the frequency of the audio signal and applies filters to remove noise. At the same time, it equalizes the audio volume level and separates multiple speakers if necessary. The preprocessed audio data (output) is used for the next transcription step.

[0179] Step 4:

[0180] Transcription of audio data

[0181] Server: The server converts the preprocessed audio data (input) into text data (output) using the Google Cloud Speech-to-Text API. Specifically, the server sends the audio data to the API and saves the returned transcription results in text format. This text data is used in the next steps of sentiment analysis and keyword extraction.

[0182] Step 5:

[0183] Sentiment analysis and keyword extraction

[0184] Server: The server performs sentiment analysis and keyword extraction on the transcribed text data (input). This is done using the spaCy and NLTK libraries. Specifically, it analyzes the text data and applies an algorithm to identify sentiment. At the same time, it processes and extracts important keywords. The analysis results (output) are used in the next performance evaluation step.

[0185] Step 6:

[0186] Performance Evaluation

[0187] Server: The server evaluates the user's performance based on the results (input) of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn or TensorFlow models that have learned from past data. Specifically, the server inputs data into the model and generates an evaluation score. The evaluation score (output) is used in the next feedback generation step.

[0188] Step 7:

[0189] Feedback Generation

[0190] Server: The server generates specific feedback based on the results of the performance evaluation (input). The generated feedback includes areas for improvement and specific advice. For example, the advice may be "You should provide more specific examples to attract customers' attention." The generated feedback (output) is provided to the user and is used in the next sending step.

[0191] Step 8:

[0192] Send and visualize feedback

[0193] Server: The server sends the generated feedback (input) to the user terminal. Specifically, the feedback data is encrypted again using the HTTPS protocol and then sent.

[0194] Terminal: The terminal receives the feedback (input) from the server and visualizes it through the user interface. The user can check the displayed feedback (output) and understand the specific improvements needed.

[0195] This is the specific flow of processing in the program for this system. In this way, advanced pre-processing of speech data and natural language processing make it possible to provide users with prompt and specific feedback.

[0196] (Application example 1)

[0197] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0198] In factory automation, the efficiency and accuracy of communication between workers and robots that give work instructions and status reports is important. However, with current systems, robots' responses to workers are often inappropriate, making it difficult to achieve natural, human-like dialogue. Therefore, in order to reduce worker stress and improve work efficiency, it is necessary to improve the robot's communication skills.

[0199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0200] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback and providing the feedback to the user, and means for analyzing and optimizing the voice data in real time so that the robot can appropriately provide work instructions and status reports. This improves the efficiency and accuracy of communication between the robot and the worker, reduces worker stress, and improves work efficiency.

[0201] "Voice data" refers to voice information collected from a user, and is the basis for analysis and processing.

[0202] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[0203] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[0204] "Text data" refers to data that has been converted from voice data into text format, and is subject to text analysis and sentiment analysis.

[0205] "Sentiment analysis" is the process of identifying emotions (e.g., joy, sadness, anger, etc.) from text data based on its expression and content.

[0206] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[0207] "Performance evaluation" is the process of evaluating the effectiveness and quality of a user's communication based on the results of sentiment analysis and keyword extraction.

[0208] "Feedback" refers to specific advice and information on areas for improvement that are provided to the user based on the evaluation results.

[0209] A "robot" is an automated machine that gives work instructions and reports on the status in factories and work sites.

[0210] "Real-time analytics" refers to the process of processing audio data instantly and making the analysis results immediately available.

[0211] "Optimization" is the process of improving a robot's communication skills and response methods to make them most effective.

[0212] The present invention is a system for optimizing interactions between robots and human workers in a factory environment, analyzing voice data in real time and generating feedback. Detailed description of the embodiments of the present invention is provided below.

[0213] System Overview

[0214] The system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends the voice data to the server. The server analyzes the voice data, generates appropriate feedback, and sends it to the terminal. The user receives the feedback and improves their communication skills based on it.

[0215] Hardware and Software

[0216] Audio recording and transmission

[0217] The device (smartphone, smart glasses, robot, etc.) is equipped with a microphone to record audio. The recorded data is encrypted for transmission to the server.

[0218] Audio data preprocessing

[0219] The server performs noise reduction and volume normalization of the audio data using the Python library "torchaudio."

[0220] Transcription of audio data

[0221] The server uses Facebook's Wav2Vec2 model to convert the audio data into text data.

[0222] Sentiment analysis and keyword extraction

[0223] For sentiment analysis and keyword extraction, we use the natural language processing library "Transformers," and also utilize models specialized for sentiment analysis.

[0224] Generate feedback

[0225] Leveraging generative AI models, it generates specific feedback based on sentiment analysis and keyword extraction, including specific improvements and recommendations.

[0226] Sending and Viewing Feedback

[0227] The feedback generated at the server is sent to the terminal and visualized through a user interface.

[0228] Specific examples

[0229] For example, consider a scenario in which a robot and a worker are interacting with each other to give new work instructions in a factory. During the work instructions, the robot records voice data in real time and sends it to a server.

[0230] The server analyzes the voice data, generates appropriate advice, and sends it back to the robot. For example, it may provide specific advice such as, "Speak more slowly so the worker can understand you." Through this process, the robot can gradually improve its communication with the worker.

[0231] Prompt Sentence Examples

[0232] You have designed a system to optimize interactions in a factory. It analyzes user voice data and provides feedback on improvements. What natural language processing model is used and how do you perform sentiment analysis?

[0233] In this way, the present invention provides a system that improves communication between robots and workers in factories, thereby increasing work efficiency, worker satisfaction, and overall business performance.

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1: Record and send your audio

[0236] The device (smartphone or robot) uses a microphone to record audio data. The recorded audio data is encrypted and sent to a server. The input here is audio data, and the output is encrypted audio data. The specific operations are pressing the recording start button and sending the data after recording is complete.

[0237] Step 2: Receiving audio data

[0238] The server receives the encrypted voice data sent from the terminal. The input is the encrypted voice data, and the output is the received voice data. The specific operation is to execute the data reception protocol on the server side.

[0239] Step 3: Preprocessing the audio data

[0240] The server performs noise removal and volume normalization on the received audio data. The input here is the received audio data, and the output is the audio data that has been noise removed and volume normalized. Specifically, the data cleansing process uses the Python library "torchaudio."

[0241] Step 4: Transcribe the audio data

[0242] The server uses Facebook's Wav2Vec2 model to convert the audio data to text. The input is the cleansed audio data, and the output is the corresponding text data. The specific operation is to input the audio data into the model and obtain the resulting text.

[0243] Step 5: Sentiment analysis and keyword extraction

[0244] The server performs sentiment analysis on the text data and extracts important keywords. The software used is "transformers." The input is transcribed text data, and the output is sentiment classification data and extracted keywords. Specific operations include inputting text data into a natural language processing model and obtaining the analysis results.

[0245] Step 6: Performance evaluation

[0246] The server evaluates the user's communication performance based on the results of sentiment analysis and keyword extraction. The input is sentiment classification data and keywords, and the output is an evaluation score. Specific operations include running the evaluation algorithm and calculating the score.

[0247] Step 7: Generate feedback

[0248] The server generates specific feedback based on the results of the performance evaluation. It uses a generative AI model. The input is the evaluation score, and the output is a specific feedback message. The specific operation is to input the generative AI model and generate the resulting message.

[0249] Step 8: Submit and view feedback

[0250] The server sends the generated feedback to the device, which then displays it to the user. The input is the specific feedback message, and the output is the feedback display on the user interface. The specific operation is the message display process on the device's UI.

[0251] The above is a detailed description of the specific processing steps of the present invention, as well as the inputs and outputs and specific operations at each step.

[0252] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0253] The present invention is a system for improving communication skills by combining voice data and an emotion engine that recognizes the user's emotions. Below, the system program is explained in detail, and its processing is described with specific examples.

[0254] System configuration

[0255] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, recognizes the user's emotions using an emotion engine, evaluates the user's performance, and provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[0256] Program processing overview

[0257] server:

[0258] 1. Receiving audio data:

[0259] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[0260] 2. Preprocessing the audio data:

[0261] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[0262] 3. Transcription of audio data:

[0263] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[0264] 4. Sentiment analysis and keyword extraction:

[0265] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[0266] 5. Applying the Emotion Engine:

[0267] The server uses an emotion engine to analyze the user's emotional trends from both voice and text data, thereby recognizing the user's emotions in real time and reflecting them in the evaluation results.

[0268] 6. Performance Evaluation:

[0269] The server scores users' performance based on sentiment analysis and keyword extraction, using an AI model trained from past data to identify patterns of high and low performers.

[0270] 7. Feedback Generation:

[0271] The server generates feedback for the user based on the results of the performance evaluation, including suggestions for improvement and specific advice. Emotional trends based on the emotion engine are also added to the feedback.

[0272] 8. Submitting Feedback:

[0273] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[0274] Device:

[0275] 1. Audio recording and transmission:

[0276] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[0277] 2. Receiving Feedback:

[0278] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[0279] User:

[0280] 1. Start recording:

[0281] The user launches the application on the terminal and starts recording business calls or sales talks.

[0282] 2. Feedback check and implementation:

[0283] Users review the feedback they receive and implement improvements in their next call or conversation.

[0284] Specific examples

[0285] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[0286] 1. Audio Recording:

[0287] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[0288] 2. Sending audio data:

[0289] After the conversation ends, the device automatically sends the recording data to the server.

[0290] 3. Data Analysis:

[0291] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data and performs sentiment analysis and keyword extraction.

[0292] 4. Applying the Emotion Engine:

[0293] The server uses an emotion engine to analyze the user's emotional trends from the voice and text data, for example, identifying the user's feelings of tension or relief when answering a customer's question.

[0294] 5. Feedback Generation:

[0295] The server scores the user's performance and generates feedback based on the results of sentiment analysis, including specific advice such as, "You seemed a little nervous when answering the customer's question. Try to be more relaxed next time."

[0296] 6. Feedback received:

[0297] The terminal receives feedback from the server and displays it to the user.

[0298] 7. Implementing improvements:

[0299] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[0300] In this way, users can utilize the emotion engine to recognize and improve their own communication skills in more detail.The system of the present invention can also significantly contribute to improving the business performance of companies.

[0301] The processing flow will be explained below.

[0302] Step 1:

[0303] User:

[0304] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[0305] Step 2:

[0306] Device:

[0307] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[0308] Step 3:

[0309] Device:

[0310] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[0311] Step 4:

[0312] server:

[0313] The server receives the voice data transmitted from the terminal.

[0314] Step 5:

[0315] server:

[0316] The server performs noise reduction, volume normalization, and speaker separation on the received audio data, improving the quality of the recording.

[0317] Step 6:

[0318] server:

[0319] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[0320] Step 7:

[0321] server:

[0322] The server uses natural language processing technology on the generated text data to perform sentiment analysis and keyword extraction, thereby clarifying the emotional tone of the conversation and key phrases.

[0323] Step 8:

[0324] server:

[0325] The server uses an emotion engine to analyze the user's emotional trends in real time from voice and text data, specifically by evaluating changes in the user's tone of voice and speaking style to recognize their emotional state.

[0326] Step 9:

[0327] server:

[0328] The server scores users' performance based on the results of sentiment analysis, keyword extraction, and emotion engine analysis. Using an AI model that has learned from past data, scores are calculated based on the patterns of high and low performers.

[0329] Step 10:

[0330] server:

[0331] The server generates specific feedback based on the results of the performance evaluation, including analysis results including sentiment trends, and suggests areas for improvement and specific advice.

[0332] Step 11:

[0333] server:

[0334] The server transmits the generated feedback to the user's terminal.

[0335] Step 12:

[0336] Device:

[0337] The terminal receives feedback from the server and visualizes it for the user to understand.

[0338] Step 13:

[0339] User:

[0340] The user can check the feedback displayed on the device and try to improve it in the next business call or sales talk. The feedback also includes specific advice on how to change emotions.

[0341] Step 14:

[0342] User:

[0343] Users can also refer to past feedback, allowing them to optimize their communication skills through a process of incremental improvement.

[0344] Example 2

[0345] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] Conventional voice data analysis systems have struggled to accurately analyze users' emotions, evaluate their performance based on the results, and provide specific feedback. Furthermore, accuracy and real-time performance issues often exist in the processes of preprocessing voice data, converting it to text data, and analyzing emotions and generating feedback. This has hindered users' ability to effectively improve their communication skills.

[0347] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data from a user terminal, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing emotion analysis and keyword extraction on the text data using natural language processing technology, means for analyzing emotion trends from both the voice data and the text data, means for evaluating performance based on the results of the emotion analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for transmitting the feedback to the user terminal. This enables accurate pre-processing of voice data, emotion recognition, real-time analysis, and provision of specific feedback.

[0348] "User terminal" refers to a device used to record and play back audio data and to communicate data with a server.

[0349] "Voice data" refers to information that is a digital recording of the user's speech.

[0350] "Server" refers to a central processing unit that processes voice data received from a user terminal and generates analysis results and feedback.

[0351] "Noise reduction" refers to the process of removing background noise from audio data to improve the clarity of the audio.

[0352] "Volume normalization" refers to the process of adjusting the volume level of audio data to a constant level to make it easier to hear.

[0353] "Text data" refers to voice data converted into text information.

[0354] "Natural language processing technology" refers to technology for analyzing natural language text and extracting structured information.

[0355] "Sentiment analysis" refers to the process of identifying and evaluating a speaker's emotional state from text or audio data.

[0356] "Keyword extraction" refers to the process of extracting important words and phrases from text data.

[0357] "Emotional trends" refer to changes or trends in emotional states over a period of time.

[0358] "Performance evaluation" refers to the process of evaluating a user's response and attitude based on the results of sentiment analysis and keyword extraction.

[0359] "Feedback" refers to suggestions for improvement or advice provided to users based on the results of performance evaluation.

[0360] "Encryption" refers to the process of converting information using a specific algorithm to make it unreadable to third parties in order to send and receive data securely.

[0361] This invention is a system that improves a user's communication skills by combining voice data and an emotion engine that recognizes the user's emotions. This system is mainly composed of a "server," a "terminal," and a "user."

[0362] The server receives the audio data from the user's device, performs noise reduction and volume normalization using audio processing libraries such as Audacity and FFmpeg, and then converts the audio data into text using a speech recognition engine. Specifically, the server uses the Google Speech-to-Text API or IBM Watson Speech to Text.

[0363] For text data, natural language processing techniques such as TextBlob and NLTK are used to perform sentiment analysis and keyword extraction. An emotion engine is used to analyze both audio and text data and identify user sentiment trends in real time. This emotion engine may be implemented using a specific software module.

[0364] Furthermore, the server evaluates the user's performance based on the results of sentiment analysis and keyword extraction using a machine learning model (using TensorFlow or PyTorch) trained from past data. Based on the evaluation results, specific feedback is generated and sent to the user's device. This feedback also includes sentiment trends.

[0365] The device records the user's business calls and sales talks and sends the audio data to a server using a recording application such as Rev Recorder. The feedback data sent from the server is visually displayed on the device's user interface.

[0366] Users simply launch the application on their device, record their business calls or sales conversations, check the feedback displayed on the device, and implement improvements in their next call or conversation. This cycle allows users to continuously improve their communication skills.

[0367] As a concrete example, consider a scenario in which a sales representative is taking on a new business deal. The sales representative converses with the client while explaining the product. During the conversation, recording begins on the device, and after the conversation ends, the audio data is sent to the server. The server processes the audio data, performs sentiment analysis and performance evaluation, and then generates feedback. For example, specific advice such as "It's good to answer customer questions in a relaxed tone" is displayed on the user's device.

[0368] The advantage of this system is that it uses an emotion engine and machine learning models to help users improve their communication skills, creating a cycle of self-improvement that also contributes to improving the business performance of the company.

[0369] Example prompt sentence:

[0370] "I want to analyze voice data and recognize emotions. Please explain the specific steps of the system that processes this data and provides feedback to the user."

[0371] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0372] Step 1:

[0373] The server receives encrypted voice data from the user's device. This voice data is a digitally recorded file of the user's speech. After receiving it, the server decrypts the data using appropriate asymmetric encryption technology and stores it in a temporary folder for internal processing.

[0374] Step 2:

[0375] The server performs noise reduction on the received audio data. This process uses an audio processing library (e.g., Audacity or FFmpeg). It takes the decoded audio data as input and applies an algorithm to remove background noise to obtain clear audio output. It also performs volume normalization to ensure that all audio data has a consistent volume level.

[0376] Step 3:

[0377] The server inputs the noise-removed and volume-normalized audio data into a speech recognition engine, which converts it into text data. Specifically, it uses the Google Speech-to-Text or IBM Watson Speech to Text API. The speech recognition engine analyzes the input audio data, converts each syllable and word into text format, and outputs it.

[0378] Step 4:

[0379] The server analyzes the generated text data using natural language processing techniques. This process uses tools such as TextBlob and NLTK to extract emotional information and keywords from the text data. The input is the text data, and the output is an emotional label (positive, negative, neutral) and a list of key keywords.

[0380] Step 5:

[0381] The server uses an emotion engine to analyze emotional trends from both voice and text data. During this process, it analyzes factors such as tone, speed, and intensity of the voice to identify the user's emotional state. It uses text data and voice features as input data and obtains real-time emotional trend information as output.

[0382] Step 6:

[0383] The server evaluates user performance based on the sentiment analysis and keyword extraction results using a machine learning model (using TensorFlow or PyTorch). This machine learning model learns from past data and identifies patterns of high and low performers. It uses the sentiment analysis results and keyword list as input data and generates a performance evaluation score as output.

[0384] Step 7:

[0385] The server generates feedback for the user based on the performance evaluation results, including specific improvements and advice. This feedback also includes the emotional trend obtained by the emotional engine. The server uses the performance evaluation score and the emotional trend information as inputs and generates a feedback message as output.

[0386] Step 8:

[0387] The server sends the generated feedback to the user's device. This data is then encrypted again and sent securely to the user's device. The device receives the feedback data and visually displays it in the user interface. The user can review this feedback and implement improvements in their next business call or sales conversation.

[0388] (Application example 2)

[0389] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0390] In modern factories, cooperation between workers and machines is extremely important. However, when workers are under high stress or tension, communication efficiency declines, which can have a negative impact on productivity and safety. Current technology lacks a mechanism for recognizing workers' emotional state in real time and providing immediate, specific feedback. Therefore, there is a need for a system that can manage workers' stress and tension, maintain smooth communication, and improve work efficiency.

[0391] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0392] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis on the text data, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for supporting communication between factory workers and machines in real time and providing feedback based on the worker's emotional state. This makes it possible to grasp the worker's emotional state in real time and provide prompt and appropriate feedback.

[0393] "Audio data" refers to data that represents audio in digital form.

[0394] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[0395] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[0396] "Text data" refers to data obtained by converting voice data into characters.

[0397] "Sentiment analysis" is a technology that identifies emotional states from text and audio data.

[0398] "Keyword extraction" is the process of extracting important words and phrases from text data.

[0399] "Performance evaluation" refers to evaluating a user's behavior and abilities based on the results of sentiment analysis and keyword extraction.

[0400] "Feedback" refers to suggestions for improvement and advice provided to users based on the evaluation results.

[0401] "Workers" refers to personnel who perform work within a factory.

[0402] "Machinery" refers to various devices and robots used within the factory.

[0403] "Real-time communication support" means instantly analyzing and supporting the dialogue between workers and machines.

[0404] "Emotional state" refers to the psychological state of a worker, such as stress or tension.

[0405] "Operation guidance" refers to instructing workers on appropriate work methods and countermeasures based on their emotional state.

[0406] The system of this invention supports real-time communication between factory workers and machines, improving work efficiency and safety. The system has three main components: a server, a terminal, and a user.

[0407] Server Functionality Description

[0408] The server is responsible for processing the received audio data. The specific processing flow is as follows:

[0409] 1. Receive audio data:

[0410] The server receives the voice data sent from the device. The received data is securely encrypted.

[0411] 2. Preprocessing the audio data:

[0412] The server performs noise reduction and volume normalization on the received audio data using pydub.

[0413] 3. Transcription of audio data:

[0414] The server uses a speech recognition model (such as Huggingface's Wav2Vec2) to convert the audio data into text.

[0415] 4. Sentiment analysis and keyword extraction:

[0416] The server then uses natural language processing technology and a sentiment analysis engine to analyze sentiment and extract keywords from the converted text data. Sentiment analysis is performed using the Huggingface pipeline.

[0417] 5. Performance evaluation and feedback generation:

[0418] The server evaluates the worker's performance based on the results of emotion analysis and keyword extraction, and generates specific feedback and operational guidance according to the worker's emotional state.

[0419] Device function description

[0420] A terminal is a device that users use in a factory. Its specific operations are as follows:

[0421] 1. Audio recording and transmission:

[0422] The device records the worker's voice in real time and sends the recorded data to a server, where it is temporarily stored and encrypted.

[0423] 2. Receiving and Viewing Feedback:

[0424] The device receives the feedback sent from the server and displays it to the user in an easy-to-understand manner, including operational guidance based on the user's emotional state.

[0425] User behavior description

[0426] Users are primarily factory workers who use the system to improve their own work. Specific examples of actions include:

[0427] 1. Start recording:

[0428] The worker starts recording the voice using the terminal at the same time as starting the work.

[0429] 2. Use feedback:

[0430] After receiving the feedback, the worker follows the instructions and implements the improvements in the next work process. For example, if tension is detected during work, the worker follows specific advice such as "take a deep breath and relax."

[0431] Examples and prompts

[0432] As a concrete example, consider the following scenario: a factory worker issues a warning about a malfunctioning machine:

[0433] Worker voice recording: "There is an abnormal noise in the machine's bearings, please check for oil leaks on the right side."

[0434] Sentiment analysis results: Detected that the worker was nervous.

[0435] Feedback: "We are in a tense situation, so please refer to the emergency response checklist for when an emergency occurs."

[0436] An example of a prompt for a generative AI model: "There is an abnormal noise in the machine's bearings, so please check the right side for oil leaks."

[0437] In this way, the system of the present invention can recognize the emotional state of workers in real time and provide appropriate feedback, thereby improving the quality of communication within the factory and significantly improving work efficiency and safety.

[0438] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0439] Step 1:

[0440] Audio recording and transmission

[0441] Description: The device records the worker's voice in real time and sends the recorded data to the server.

[0442] Input: Worker's voice input.

[0443] Output: Encrypted audio data.

[0444] How it works: The device records the worker's voice in real time and temporarily stores the data. The stored data is encrypted and sent to a server using a secure protocol (e.g., HTTPS).

[0445] Step 2:

[0446] Audio data preprocessing

[0447] Description: The server performs noise reduction and volume normalization on the received audio data.

[0448] Input: Encrypted audio data received from the device.

[0449] Output: Denoised and volume-normalised audio data.

[0450] Specific operation: The server decrypts the received audio data based on the encryption method, and uses the pydub library to perform noise removal and volume normalization.

[0451] Step 3:

[0452] Transcription of audio data

[0453] Description: The server uses a speech recognition model to convert the audio data into text data.

[0454] Input: Denoised and volume-normalised audio data.

[0455] Output: Text data.

[0456] Specific operation: The server inputs the noise-removed and volume-normalized audio data into a speech recognition model (e.g., Huggingface's Wav2Vec2) and converts the audio into corresponding text data.

[0457] Step 4:

[0458] Sentiment analysis and keyword extraction

[0459] Description: The server performs sentiment analysis and keyword extraction on the converted text data.

[0460] Input: Transcribed text data.

[0461] Output: Emotional state and keyword extraction results.

[0462] Specific operation: The server inputs text data into the Huggingface pipeline and performs sentiment analysis and keyword extraction. The obtained emotional state and keywords are used to evaluate performance in the next step.

[0463] Step 5:

[0464] Performance evaluation and feedback generation

[0465] Description: The server evaluates the worker's performance based on the results of sentiment analysis and keyword extraction, and generates specific feedback and operation guidance.

[0466] Input: Emotional state and keyword extraction results.

[0467] Output: Specific feedback and operation guidance.

[0468] Specific Action: The server integrates the results of sentiment analysis and keyword extraction to evaluate the worker's overall performance, including stress and tension. The generated feedback includes specific improvements and operational guidance.

[0469] Step 6:

[0470] Receiving and viewing feedback

[0471] Description: The device receives feedback sent from the server and displays it visually to the user.

[0472] Input: Feedback received from the server.

[0473] Output: Feedback information that is displayed to the user.

[0474] Specific operation: The device asynchronously receives feedback sent from the server and displays it in an easy-to-understand visualization format for the user, allowing the user to check work improvements and emergency response measures in real time.

[0475] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0476] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0477] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0478] [Second embodiment]

[0479] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0480] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0481] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0482] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0483] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0484] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0485] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0486] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0487] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0488] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0489] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0490] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0491] The present invention is a system that receives voice data and analyzes it to improve a user's communication skills. The system program will be explained below, and its processing will be described with specific examples.

[0492] System Overview

[0493] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, evaluates the performance, and then provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[0494] Program processing overview

[0495] server:

[0496] 1. Receiving audio data:

[0497] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[0498] 2. Preprocessing the audio data:

[0499] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[0500] 3. Transcription of audio data:

[0501] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[0502] 4. Sentiment analysis and keyword extraction:

[0503] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[0504] 5. Performance Evaluation:

[0505] The server scores users' performance based on sentiment analysis and keyword extraction. The AI ​​model learns from past data and identifies patterns of high and low performers.

[0506] 6. Feedback Generation:

[0507] The server generates specific feedback based on the results of the performance evaluation, including areas for improvement and specific advice.

[0508] 7. Submitting Feedback:

[0509] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[0510] Device:

[0511] 1. Audio recording and transmission:

[0512] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[0513] 2. Receiving Feedback:

[0514] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[0515] User:

[0516] 1. Start recording:

[0517] The user launches the application on the terminal and starts recording business calls or sales talks.

[0518] 2. Feedback check and implementation:

[0519] Users review the feedback they receive and implement improvements in their next call or conversation.

[0520] Specific examples

[0521] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[0522] 1. Audio Recording:

[0523] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[0524] 2. Sending audio data:

[0525] After the conversation ends, the device automatically sends the recording data to the server.

[0526] 3. Data analysis and feedback generation:

[0527] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data, performs sentiment analysis and keyword extraction, extracts points that need emphasis during the conversation, and generates feedback.

[0528] 4. Feedback received:

[0529] The device receives feedback from the server and provides specific advice such as "ask more open-ended questions to further draw out the customer's concerns."

[0530] 5. Implement improvements:

[0531] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[0532] In this way, users can continuously optimize their communication skills through data-driven coaching using AI. The system of the present invention can also significantly contribute to improving a company's business performance.

[0533] The processing flow will be explained below.

[0534] Step 1:

[0535] User:

[0536] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[0537] Step 2:

[0538] Device:

[0539] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[0540] Step 3:

[0541] Device:

[0542] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[0543] Step 4:

[0544] server:

[0545] The server receives the voice data transmitted from the terminal.

[0546] Step 5:

[0547] server:

[0548] The server performs noise reduction and volume normalization on the received audio data, improving the sound quality of the recorded data.

[0549] Step 6:

[0550] server:

[0551] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[0552] Step 7:

[0553] server:

[0554] The server uses natural language processing technology to perform sentiment analysis on the generated text data, classifying the emotional tone of the utterance as positive, negative, neutral, etc.

[0555] Step 8:

[0556] server:

[0557] The server simultaneously extracts keywords and important phrases from the text data, making the main points of the conversation clearer.

[0558] Step 9:

[0559] server:

[0560] The server evaluates user performance based on text data, sentiment analysis, and keyword extraction. Using an AI model trained on past data, it scores users based on the patterns of high and low performers.

[0561] Step 10:

[0562] server:

[0563] Based on the results of the performance evaluation, the server generates feedback to the user, including areas for improvement and specific advice.

[0564] Step 11:

[0565] server:

[0566] The server transmits the generated feedback to the user's terminal.

[0567] Step 12:

[0568] Device:

[0569] The terminal receives feedback from the server and visualizes it for the user to understand.

[0570] Step 13:

[0571] User:

[0572] The user can check the feedback displayed on the device and use it to make improvements in their next business call or sales conversation.

[0573] Example 1

[0574] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0575] Conventional systems for improving communication skills that use voice data have the problem of being unable to consistently perform advanced processing such as noise removal, volume normalization, emotion analysis, and keyword extraction when analyzing user voice data. Furthermore, the inability to provide real-time evaluation or feedback makes it difficult to quickly improve users' communication skills.

[0576] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0577] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis and keyword extraction on the text data, means for evaluating performance based on the results of the sentiment analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for visualizing the feedback through a user interface. This makes it possible to perform advanced voice data preprocessing and natural language processing in real time and provide specific feedback to the user.

[0578] "Audio data" refers to data in which audio is recorded in digital format.

[0579] A "receiving means" is a device or program that has the function of receiving specific data from another device or system.

[0580] "Noise reduction" is the process of reducing or eliminating unwanted background noise in audio data.

[0581] "Volume normalization" is a process for making the volume level of audio data uniform.

[0582] "Means for converting to text data" refers to a system or algorithm for converting audio data into a corresponding string of characters.

[0583] "Sentiment analysis" is the process of analyzing and extracting the type and intensity of emotions from text data.

[0584] "Keyword extraction" is a technique for extracting important words and phrases from text data.

[0585] A "means for evaluating performance" is a device or program that has the function of evaluating a user's performance or results based on collected data.

[0586] "Feedback" refers to specific advice or suggestions provided to the user based on the evaluation results.

[0587] A "user interface" is the visual or operational means by which a user interacts with a system.

[0588] The present invention is a system that receives and analyzes voice data to improve a user's communication skills. This system is mainly composed of a "server," a "terminal," and a "user."

[0589] Hardware and Software Overview

[0590] server:

[0591] The server plays a central role in processing the audio data. The following software and tools are used for this processing:

[0592] Audio processing library: Librosa

[0593] Speech recognition engine: Google Cloud Speech-to-Text API

[0594] Natural language processing libraries: spaCy, NLTK

[0595] Machine learning libraries: Scikit-learn, TensorFlow

[0596] Device:

[0597] The terminal is a device that allows users to record audio and send and receive data to and from the server. Typical terminals are smartphones or tablets with built-in recording applications.

[0598] Processing flow and specific examples

[0599] Audio recording and transmission

[0600] The user starts the recording application on the device and records business calls or sales talks. The audio is temporarily stored on the device, and once recording is complete, it is encrypted and sent to the server.

[0601] Receiving and preprocessing audio data

[0602] The server receives the encrypted audio data, which is then denoised and normalised using the Librosa library, improving the quality of the audio data.

[0603] Conversion to text data

[0604] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data, which is then used for sentiment analysis and keyword extraction.

[0605] Sentiment analysis and keyword extraction

[0606] The server analyzes emotions from the transcribed text data and extracts important keywords. This process uses the spaCy and NLTK libraries. For example, it determines whether the emotion contained in the text data is "joy" or "anger."

[0607] Performance evaluation and feedback generation

[0608] The server evaluates the user's performance based on the results of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn and TensorFlow models trained on past data. Specific feedback is generated based on the evaluation results. For example, advice such as "You should ask more open-ended questions to further elicit customer concerns" is provided.

[0609] Send and visualize feedback

[0610] The server transmits the generated feedback to the user's terminal, which receives the feedback, visualizes it through a user interface, and presents it to the user.

[0611] Specific use cases

[0612] For example, the following scenario shows how a sales representative might use the system when approaching a new business deal.

[0613] 1. The sales representative (user) starts recording on the device while talking to the client.

[0614] 2. When the conversation ends, the device encrypts the recording and sends it to the server.

[0615] 3. The server receives the audio data and performs preprocessing, converting the audio into text data, and performing sentiment analysis and keyword extraction.

[0616] 4. The server generates feedback such as "You should simplify the explanations in areas where the customer is not interested."

[0617] 5. The device receives this feedback and presents it to the user in a visual form.

[0618] 6. The salesperson (user) uses this feedback in the next sales meeting to improve communication.

[0619] Prompt Sentence Examples

[0620] "Please analyze the audio data of this sales pitch and extract user sentiment and keywords. Please also include areas for improvement and specific advice in your feedback."

[0621] "Perform sentiment analysis on the following text data, extract keywords, and generate feedback: 'The customer asked many questions during the product explanation, which we answered well. However, you should ask more open-ended questions.'"

[0622] The above is a specific embodiment for carrying out the present invention. This system allows users to receive feedback in real time, enabling them to quickly improve their communication skills.

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] Audio recording and transmission

[0626] Device: The user launches the device's recording application and records a business call or sales conversation. Specifically, the device uses the built-in microphone to collect voice data (input) and temporarily stores it within the application. After the recording is complete, the voice data is encrypted. The encrypted voice data (output) is sent to the server using the HTTPS protocol.

[0627] Step 2:

[0628] Receiving audio data

[0629] Server: The server receives the encrypted voice data (input) sent from the device. Specifically, the server receives the data using the HTTPS protocol and stores it securely in a database. The stored voice data (output) is used for the next preprocessing step.

[0630] Step 3:

[0631] Audio data preprocessing

[0632] Server: The server performs noise reduction, volume normalization, and speaker separation on the stored audio data (input). This is done using the Librosa library. Specifically, the server analyzes the frequency of the audio signal and applies filters to remove noise. At the same time, it equalizes the audio volume level and separates multiple speakers if necessary. The preprocessed audio data (output) is used for the next transcription step.

[0633] Step 4:

[0634] Transcription of audio data

[0635] Server: The server converts the preprocessed audio data (input) into text data (output) using the Google Cloud Speech-to-Text API. Specifically, the server sends the audio data to the API and saves the returned transcription results in text format. This text data is used in the next steps of sentiment analysis and keyword extraction.

[0636] Step 5:

[0637] Sentiment analysis and keyword extraction

[0638] Server: The server performs sentiment analysis and keyword extraction on the transcribed text data (input). This is done using the spaCy and NLTK libraries. Specifically, it analyzes the text data and applies an algorithm to identify sentiment. At the same time, it processes and extracts important keywords. The analysis results (output) are used in the next performance evaluation step.

[0639] Step 6:

[0640] Performance Evaluation

[0641] Server: The server evaluates the user's performance based on the results (input) of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn or TensorFlow models that have learned from past data. Specifically, the server inputs data into the model and generates an evaluation score. The evaluation score (output) is used in the next feedback generation step.

[0642] Step 7:

[0643] Feedback Generation

[0644] Server: The server generates specific feedback based on the results of the performance evaluation (input). The generated feedback includes areas for improvement and specific advice. For example, the advice may be "You should provide more specific examples to attract customers' attention." The generated feedback (output) is provided to the user and is used in the next sending step.

[0645] Step 8:

[0646] Send and visualize feedback

[0647] Server: The server sends the generated feedback (input) to the user terminal. Specifically, the feedback data is encrypted again using the HTTPS protocol and then sent.

[0648] Terminal: The terminal receives the feedback (input) from the server and visualizes it through the user interface. The user can check the displayed feedback (output) and understand the specific improvements needed.

[0649] This is the specific flow of processing in the program for this system. In this way, advanced pre-processing of speech data and natural language processing make it possible to provide users with prompt and specific feedback.

[0650] (Application example 1)

[0651] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0652] In factory automation, the efficiency and accuracy of communication between workers and robots that give work instructions and status reports is important. However, with current systems, robots' responses to workers are often inappropriate, making it difficult to achieve natural, human-like dialogue. Therefore, in order to reduce worker stress and improve work efficiency, it is necessary to improve the robot's communication skills.

[0653] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0654] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback and providing the feedback to the user, and means for analyzing and optimizing the voice data in real time so that the robot can appropriately provide work instructions and status reports. This improves the efficiency and accuracy of communication between the robot and the worker, reduces worker stress, and improves work efficiency.

[0655] "Voice data" refers to voice information collected from a user, and is the basis for analysis and processing.

[0656] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[0657] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[0658] "Text data" refers to data that has been converted from voice data into text format, and is subject to text analysis and sentiment analysis.

[0659] "Sentiment analysis" is the process of identifying emotions (e.g., joy, sadness, anger, etc.) from text data based on its expression and content.

[0660] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[0661] "Performance evaluation" is the process of evaluating the effectiveness and quality of a user's communication based on the results of sentiment analysis and keyword extraction.

[0662] "Feedback" refers to specific advice and information on areas for improvement that are provided to the user based on the evaluation results.

[0663] A "robot" is an automated machine that gives work instructions and reports on the status in factories and work sites.

[0664] "Real-time analytics" refers to the process of processing audio data instantly and making the analysis results immediately available.

[0665] "Optimization" is the process of improving a robot's communication skills and response methods to make them most effective.

[0666] The present invention is a system for optimizing interactions between robots and human workers in a factory environment, analyzing voice data in real time and generating feedback. Detailed description of the embodiments of the present invention is provided below.

[0667] System Overview

[0668] The system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends the voice data to the server. The server analyzes the voice data, generates appropriate feedback, and sends it to the terminal. The user receives the feedback and improves their communication skills based on it.

[0669] Hardware and Software

[0670] Audio recording and transmission

[0671] The device (smartphone, smart glasses, robot, etc.) is equipped with a microphone to record audio. The recorded data is encrypted for transmission to the server.

[0672] Audio data preprocessing

[0673] The server performs noise reduction and volume normalization of the audio data using the Python library "torchaudio."

[0674] Transcription of audio data

[0675] The server uses Facebook's Wav2Vec2 model to convert the audio data into text data.

[0676] Sentiment analysis and keyword extraction

[0677] For sentiment analysis and keyword extraction, we use the natural language processing library "Transformers," and also utilize models specialized for sentiment analysis.

[0678] Generate feedback

[0679] Leveraging generative AI models, it generates specific feedback based on sentiment analysis and keyword extraction, including specific improvements and recommendations.

[0680] Sending and Viewing Feedback

[0681] The feedback generated at the server is sent to the terminal and visualized through a user interface.

[0682] Specific examples

[0683] For example, consider a scenario in which a robot and a worker are interacting with each other to give new work instructions in a factory. During the work instructions, the robot records voice data in real time and sends it to a server.

[0684] The server analyzes the voice data, generates appropriate advice, and sends it back to the robot. For example, it may provide specific advice such as, "Speak more slowly so the worker can understand you." Through this process, the robot can gradually improve its communication with the worker.

[0685] Prompt Sentence Examples

[0686] You have designed a system to optimize interactions in a factory. It analyzes user voice data and provides feedback on improvements. What natural language processing model is used and how do you perform sentiment analysis?

[0687] In this way, the present invention provides a system that improves communication between robots and workers in factories, thereby increasing work efficiency, worker satisfaction, and overall business performance.

[0688] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0689] Step 1: Record and send your audio

[0690] The device (smartphone or robot) uses a microphone to record audio data. The recorded audio data is encrypted and sent to a server. The input here is audio data, and the output is encrypted audio data. The specific operations are pressing the recording start button and sending the data after recording is complete.

[0691] Step 2: Receiving audio data

[0692] The server receives the encrypted voice data sent from the terminal. The input is the encrypted voice data, and the output is the received voice data. The specific operation is to execute the data reception protocol on the server side.

[0693] Step 3: Preprocessing the audio data

[0694] The server performs noise removal and volume normalization on the received audio data. The input here is the received audio data, and the output is the audio data that has been noise removed and volume normalized. Specifically, the data cleansing process uses the Python library "torchaudio."

[0695] Step 4: Transcribe the audio data

[0696] The server uses Facebook's Wav2Vec2 model to convert the audio data to text. The input is the cleansed audio data, and the output is the corresponding text data. The specific operation is to input the audio data into the model and obtain the resulting text.

[0697] Step 5: Sentiment analysis and keyword extraction

[0698] The server performs sentiment analysis on the text data and extracts important keywords. The software used is "transformers." The input is transcribed text data, and the output is sentiment classification data and extracted keywords. Specific operations include inputting text data into a natural language processing model and obtaining the analysis results.

[0699] Step 6: Performance evaluation

[0700] The server evaluates the user's communication performance based on the results of sentiment analysis and keyword extraction. The input is sentiment classification data and keywords, and the output is an evaluation score. Specific operations include running the evaluation algorithm and calculating the score.

[0701] Step 7: Generate feedback

[0702] The server generates specific feedback based on the results of the performance evaluation. It uses a generative AI model. The input is the evaluation score, and the output is a specific feedback message. The specific operation is to input the generative AI model and generate the resulting message.

[0703] Step 8: Submit and view feedback

[0704] The server sends the generated feedback to the device, which then displays it to the user. The input is the specific feedback message, and the output is the feedback display on the user interface. The specific operation is the message display process on the device's UI.

[0705] The above is a detailed description of the specific processing steps of the present invention, as well as the inputs and outputs and specific operations at each step.

[0706] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0707] The present invention is a system for improving communication skills by combining voice data and an emotion engine that recognizes the user's emotions. Below, the system program is explained in detail, and its processing is described with specific examples.

[0708] System configuration

[0709] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, recognizes the user's emotions using an emotion engine, evaluates the user's performance, and provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[0710] Program processing overview

[0711] server:

[0712] 1. Receiving audio data:

[0713] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[0714] 2. Preprocessing the audio data:

[0715] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[0716] 3. Transcription of audio data:

[0717] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[0718] 4. Sentiment analysis and keyword extraction:

[0719] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[0720] 5. Applying the Emotion Engine:

[0721] The server uses an emotion engine to analyze the user's emotional trends from both voice and text data, thereby recognizing the user's emotions in real time and reflecting them in the evaluation results.

[0722] 6. Performance Evaluation:

[0723] The server scores users' performance based on sentiment analysis and keyword extraction, using an AI model trained from past data to identify patterns of high and low performers.

[0724] 7. Feedback Generation:

[0725] The server generates feedback for the user based on the results of the performance evaluation, including suggestions for improvement and specific advice. Emotional trends based on the emotion engine are also added to the feedback.

[0726] 8. Submitting Feedback:

[0727] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[0728] Device:

[0729] 1. Audio recording and transmission:

[0730] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[0731] 2. Receiving Feedback:

[0732] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[0733] User:

[0734] 1. Start recording:

[0735] The user launches the application on the terminal and starts recording business calls or sales talks.

[0736] 2. Feedback check and implementation:

[0737] Users review the feedback they receive and implement improvements in their next call or conversation.

[0738] Specific examples

[0739] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[0740] 1. Audio Recording:

[0741] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[0742] 2. Sending audio data:

[0743] After the conversation ends, the device automatically sends the recording data to the server.

[0744] 3. Data Analysis:

[0745] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data and performs sentiment analysis and keyword extraction.

[0746] 4. Applying the Emotion Engine:

[0747] The server uses an emotion engine to analyze the user's emotional trends from the voice and text data, for example, identifying the user's feelings of tension or relief when answering a customer's question.

[0748] 5. Feedback Generation:

[0749] The server scores the user's performance and generates feedback based on the results of sentiment analysis, including specific advice such as, "You seemed a little nervous when answering the customer's question. Try to be more relaxed next time."

[0750] 6. Feedback received:

[0751] The terminal receives feedback from the server and displays it to the user.

[0752] 7. Implementing improvements:

[0753] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[0754] In this way, users can utilize the emotion engine to recognize and improve their own communication skills in more detail.The system of the present invention can also significantly contribute to improving the business performance of companies.

[0755] The processing flow will be explained below.

[0756] Step 1:

[0757] User:

[0758] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[0759] Step 2:

[0760] Device:

[0761] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[0762] Step 3:

[0763] Device:

[0764] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[0765] Step 4:

[0766] server:

[0767] The server receives the voice data transmitted from the terminal.

[0768] Step 5:

[0769] server:

[0770] The server performs noise reduction, volume normalization, and speaker separation on the received audio data, improving the quality of the recording.

[0771] Step 6:

[0772] server:

[0773] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[0774] Step 7:

[0775] server:

[0776] The server uses natural language processing technology on the generated text data to perform sentiment analysis and keyword extraction, thereby clarifying the emotional tone of the conversation and key phrases.

[0777] Step 8:

[0778] server:

[0779] The server uses an emotion engine to analyze the user's emotional trends in real time from voice and text data, specifically by evaluating changes in the user's tone of voice and speaking style to recognize their emotional state.

[0780] Step 9:

[0781] server:

[0782] The server scores users' performance based on the results of sentiment analysis, keyword extraction, and emotion engine analysis. Using an AI model that has learned from past data, scores are calculated based on the patterns of high and low performers.

[0783] Step 10:

[0784] server:

[0785] The server generates specific feedback based on the results of the performance evaluation, including analysis results including sentiment trends, and suggests areas for improvement and specific advice.

[0786] Step 11:

[0787] server:

[0788] The server transmits the generated feedback to the user's terminal.

[0789] Step 12:

[0790] Device:

[0791] The terminal receives feedback from the server and visualizes it for the user to understand.

[0792] Step 13:

[0793] User:

[0794] The user can check the feedback displayed on the device and try to improve it in the next business call or sales talk. The feedback also includes specific advice on how to change emotions.

[0795] Step 14:

[0796] User:

[0797] Users can also refer to past feedback, allowing them to optimize their communication skills through a process of incremental improvement.

[0798] Example 2

[0799] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0800] Conventional voice data analysis systems have struggled to accurately analyze users' emotions, evaluate their performance based on the results, and provide specific feedback. Furthermore, accuracy and real-time performance issues often exist in the processes of preprocessing voice data, converting it to text data, and analyzing emotions and generating feedback. This has hindered users' ability to effectively improve their communication skills.

[0801] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data from a user terminal, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing emotion analysis and keyword extraction on the text data using natural language processing technology, means for analyzing emotion trends from both the voice data and the text data, means for evaluating performance based on the results of the emotion analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for transmitting the feedback to the user terminal. This enables accurate pre-processing of voice data, emotion recognition, real-time analysis, and provision of specific feedback.

[0802] "User terminal" refers to a device used to record and play back audio data and to communicate data with a server.

[0803] "Voice data" refers to information that is a digital recording of the user's speech.

[0804] "Server" refers to a central processing unit that processes voice data received from a user terminal and generates analysis results and feedback.

[0805] "Noise reduction" refers to the process of removing background noise from audio data to improve the clarity of the audio.

[0806] "Volume normalization" refers to the process of adjusting the volume level of audio data to a constant level to make it easier to hear.

[0807] "Text data" refers to voice data converted into text information.

[0808] "Natural language processing technology" refers to technology for analyzing natural language text and extracting structured information.

[0809] "Sentiment analysis" refers to the process of identifying and evaluating a speaker's emotional state from text or audio data.

[0810] "Keyword extraction" refers to the process of extracting important words and phrases from text data.

[0811] "Emotional trends" refer to changes or trends in emotional states over a period of time.

[0812] "Performance evaluation" refers to the process of evaluating a user's response and attitude based on the results of sentiment analysis and keyword extraction.

[0813] "Feedback" refers to suggestions for improvement or advice provided to users based on the results of performance evaluation.

[0814] "Encryption" refers to the process of converting information using a specific algorithm to make it unreadable to third parties in order to send and receive data securely.

[0815] This invention is a system that improves a user's communication skills by combining voice data and an emotion engine that recognizes the user's emotions. This system is mainly composed of a "server," a "terminal," and a "user."

[0816] The server receives the audio data from the user's device, performs noise reduction and volume normalization using audio processing libraries such as Audacity and FFmpeg, and then converts the audio data into text using a speech recognition engine. Specifically, the server uses the Google Speech-to-Text API or IBM Watson Speech to Text.

[0817] For text data, natural language processing techniques such as TextBlob and NLTK are used to perform sentiment analysis and keyword extraction. An emotion engine is used to analyze both audio and text data and identify user sentiment trends in real time. This emotion engine may be implemented using a specific software module.

[0818] Furthermore, the server evaluates the user's performance based on the results of sentiment analysis and keyword extraction using a machine learning model (using TensorFlow or PyTorch) trained from past data. Based on the evaluation results, specific feedback is generated and sent to the user's device. This feedback also includes sentiment trends.

[0819] The device records the user's business calls and sales talks and sends the audio data to a server using a recording application such as Rev Recorder. The feedback data sent from the server is visually displayed on the device's user interface.

[0820] Users simply launch the application on their device, record their business calls or sales conversations, check the feedback displayed on the device, and implement improvements in their next call or conversation. This cycle allows users to continuously improve their communication skills.

[0821] As a concrete example, consider a scenario in which a sales representative is taking on a new business deal. The sales representative converses with the client while explaining the product. During the conversation, recording begins on the device, and after the conversation ends, the audio data is sent to the server. The server processes the audio data, performs sentiment analysis and performance evaluation, and then generates feedback. For example, specific advice such as "It's good to answer customer questions in a relaxed tone" is displayed on the user's device.

[0822] The advantage of this system is that it uses an emotion engine and machine learning models to help users improve their communication skills, creating a cycle of self-improvement that also contributes to improving the business performance of the company.

[0823] Example prompt sentence:

[0824] "I want to analyze voice data and recognize emotions. Please explain the specific steps of the system that processes this data and provides feedback to the user."

[0825] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0826] Step 1:

[0827] The server receives encrypted voice data from the user's device. This voice data is a digitally recorded file of the user's speech. After receiving it, the server decrypts the data using appropriate asymmetric encryption technology and stores it in a temporary folder for internal processing.

[0828] Step 2:

[0829] The server performs noise reduction on the received audio data. This process uses an audio processing library (e.g., Audacity or FFmpeg). It takes the decoded audio data as input and applies an algorithm to remove background noise to obtain clear audio output. It also performs volume normalization to ensure that all audio data has a consistent volume level.

[0830] Step 3:

[0831] The server inputs the noise-removed and volume-normalized audio data into a speech recognition engine, which converts it into text data. Specifically, it uses the Google Speech-to-Text or IBM Watson Speech to Text API. The speech recognition engine analyzes the input audio data, converts each syllable and word into text format, and outputs it.

[0832] Step 4:

[0833] The server analyzes the generated text data using natural language processing techniques. This process uses tools such as TextBlob and NLTK to extract emotional information and keywords from the text data. The input is the text data, and the output is an emotional label (positive, negative, neutral) and a list of key keywords.

[0834] Step 5:

[0835] The server uses an emotion engine to analyze emotional trends from both voice and text data. During this process, it analyzes factors such as tone, speed, and intensity of the voice to identify the user's emotional state. It uses text data and voice features as input data and obtains real-time emotional trend information as output.

[0836] Step 6:

[0837] The server evaluates user performance based on the sentiment analysis and keyword extraction results using a machine learning model (using TensorFlow or PyTorch). This machine learning model learns from past data and identifies patterns of high and low performers. It uses the sentiment analysis results and keyword list as input data and generates a performance evaluation score as output.

[0838] Step 7:

[0839] The server generates feedback for the user based on the performance evaluation results, including specific improvements and advice. This feedback also includes the emotional trend obtained by the emotional engine. The server uses the performance evaluation score and the emotional trend information as inputs and generates a feedback message as output.

[0840] Step 8:

[0841] The server sends the generated feedback to the user's device. This data is then encrypted again and sent securely to the user's device. The device receives the feedback data and visually displays it in the user interface. The user can review this feedback and implement improvements in their next business call or sales conversation.

[0842] (Application example 2)

[0843] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0844] In modern factories, cooperation between workers and machines is extremely important. However, when workers are under high stress or tension, communication efficiency declines, which can have a negative impact on productivity and safety. Current technology lacks a mechanism for recognizing workers' emotional state in real time and providing immediate, specific feedback. Therefore, there is a need for a system that can manage workers' stress and tension, maintain smooth communication, and improve work efficiency.

[0845] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0846] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis on the text data, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for supporting communication between factory workers and machines in real time and providing feedback based on the worker's emotional state. This makes it possible to grasp the worker's emotional state in real time and provide prompt and appropriate feedback.

[0847] "Audio data" refers to data that represents audio in digital form.

[0848] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[0849] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[0850] "Text data" refers to data obtained by converting voice data into characters.

[0851] "Sentiment analysis" is a technology that identifies emotional states from text and audio data.

[0852] "Keyword extraction" is the process of extracting important words and phrases from text data.

[0853] "Performance evaluation" refers to evaluating a user's behavior and abilities based on the results of sentiment analysis and keyword extraction.

[0854] "Feedback" refers to suggestions for improvement and advice provided to users based on the evaluation results.

[0855] "Workers" refers to personnel who perform work within a factory.

[0856] "Machinery" refers to various devices and robots used within the factory.

[0857] "Real-time communication support" means instantly analyzing and supporting the dialogue between workers and machines.

[0858] "Emotional state" refers to the psychological state of a worker, such as stress or tension.

[0859] "Operation guidance" refers to instructing workers on appropriate work methods and countermeasures based on their emotional state.

[0860] The system of this invention supports real-time communication between factory workers and machines, improving work efficiency and safety. The system has three main components: a server, a terminal, and a user.

[0861] Server Functionality Description

[0862] The server is responsible for processing the received audio data. The specific processing flow is as follows:

[0863] 1. Receive audio data:

[0864] The server receives the voice data sent from the device. The received data is securely encrypted.

[0865] 2. Preprocessing the audio data:

[0866] The server performs noise reduction and volume normalization on the received audio data using pydub.

[0867] 3. Transcription of audio data:

[0868] The server uses a speech recognition model (such as Huggingface's Wav2Vec2) to convert the audio data into text.

[0869] 4. Sentiment analysis and keyword extraction:

[0870] The server then uses natural language processing technology and a sentiment analysis engine to analyze sentiment and extract keywords from the converted text data. Sentiment analysis is performed using the Huggingface pipeline.

[0871] 5. Performance evaluation and feedback generation:

[0872] The server evaluates the worker's performance based on the results of emotion analysis and keyword extraction, and generates specific feedback and operational guidance according to the worker's emotional state.

[0873] Device function description

[0874] A terminal is a device that users use in a factory. Its specific operations are as follows:

[0875] 1. Audio recording and transmission:

[0876] The device records the worker's voice in real time and sends the recorded data to a server, where it is temporarily stored and encrypted.

[0877] 2. Receiving and Viewing Feedback:

[0878] The device receives the feedback sent from the server and displays it to the user in an easy-to-understand manner, including operational guidance based on the user's emotional state.

[0879] User behavior description

[0880] Users are primarily factory workers who use the system to improve their own work. Specific examples of actions include:

[0881] 1. Start recording:

[0882] The worker starts recording the voice using the terminal at the same time as starting the work.

[0883] 2. Use feedback:

[0884] After receiving the feedback, the worker follows the instructions and implements the improvements in the next work process. For example, if tension is detected during work, the worker follows specific advice such as "take a deep breath and relax."

[0885] Examples and prompts

[0886] As a concrete example, consider the following scenario: a factory worker issues a warning about a malfunctioning machine:

[0887] Worker voice recording: "There is an abnormal noise in the machine's bearings, please check for oil leaks on the right side."

[0888] Sentiment analysis results: Detected that the worker was nervous.

[0889] Feedback: "We are in a tense situation, so please refer to the emergency response checklist for when an emergency occurs."

[0890] An example of a prompt for a generative AI model: "There is an abnormal noise in the machine's bearings, so please check the right side for oil leaks."

[0891] In this way, the system of the present invention can recognize the emotional state of workers in real time and provide appropriate feedback, thereby improving the quality of communication within the factory and significantly improving work efficiency and safety.

[0892] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0893] Step 1:

[0894] Audio recording and transmission

[0895] Description: The device records the worker's voice in real time and sends the recorded data to the server.

[0896] Input: Worker's voice input.

[0897] Output: Encrypted audio data.

[0898] How it works: The device records the worker's voice in real time and temporarily stores the data. The stored data is encrypted and sent to a server using a secure protocol (e.g., HTTPS).

[0899] Step 2:

[0900] Audio data preprocessing

[0901] Description: The server performs noise reduction and volume normalization on the received audio data.

[0902] Input: Encrypted audio data received from the device.

[0903] Output: Denoised and volume-normalised audio data.

[0904] Specific operation: The server decrypts the received audio data based on the encryption method, and uses the pydub library to perform noise removal and volume normalization.

[0905] Step 3:

[0906] Transcription of audio data

[0907] Description: The server uses a speech recognition model to convert the audio data into text data.

[0908] Input: Denoised and volume-normalised audio data.

[0909] Output: Text data.

[0910] Specific operation: The server inputs the noise-removed and volume-normalized audio data into a speech recognition model (e.g., Huggingface's Wav2Vec2) and converts the audio into corresponding text data.

[0911] Step 4:

[0912] Sentiment analysis and keyword extraction

[0913] Description: The server performs sentiment analysis and keyword extraction on the converted text data.

[0914] Input: Transcribed text data.

[0915] Output: Emotional state and keyword extraction results.

[0916] Specific operation: The server inputs text data into the Huggingface pipeline and performs sentiment analysis and keyword extraction. The obtained emotional state and keywords are used to evaluate performance in the next step.

[0917] Step 5:

[0918] Performance evaluation and feedback generation

[0919] Description: The server evaluates the worker's performance based on the results of sentiment analysis and keyword extraction, and generates specific feedback and operation guidance.

[0920] Input: Emotional state and keyword extraction results.

[0921] Output: Specific feedback and operation guidance.

[0922] Specific Action: The server integrates the results of sentiment analysis and keyword extraction to evaluate the worker's overall performance, including stress and tension. The generated feedback includes specific improvements and operational guidance.

[0923] Step 6:

[0924] Receiving and viewing feedback

[0925] Description: The device receives feedback sent from the server and displays it visually to the user.

[0926] Input: Feedback received from the server.

[0927] Output: Feedback information that is displayed to the user.

[0928] Specific operation: The device asynchronously receives feedback sent from the server and displays it in an easy-to-understand visualization format for the user, allowing the user to check work improvements and emergency response measures in real time.

[0929] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0930] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0931] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0932] [Third embodiment]

[0933] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0934] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0935] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0936] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0937] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0938] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0939] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0940] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0941] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0942] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0943] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0944] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0945] The present invention is a system that receives voice data and analyzes it to improve a user's communication skills. The system program will be explained below, and its processing will be described with specific examples.

[0946] System Overview

[0947] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, evaluates the performance, and then provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[0948] Program processing overview

[0949] server:

[0950] 1. Receiving audio data:

[0951] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[0952] 2. Preprocessing the audio data:

[0953] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[0954] 3. Transcription of audio data:

[0955] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[0956] 4. Sentiment analysis and keyword extraction:

[0957] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[0958] 5. Performance Evaluation:

[0959] The server scores users' performance based on sentiment analysis and keyword extraction. The AI ​​model learns from past data and identifies patterns of high and low performers.

[0960] 6. Feedback Generation:

[0961] The server generates specific feedback based on the results of the performance evaluation, including areas for improvement and specific advice.

[0962] 7. Submitting Feedback:

[0963] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[0964] Device:

[0965] 1. Audio recording and transmission:

[0966] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[0967] 2. Receiving Feedback:

[0968] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[0969] User:

[0970] 1. Start recording:

[0971] The user launches the application on the terminal and starts recording business calls or sales talks.

[0972] 2. Feedback check and implementation:

[0973] Users review the feedback they receive and implement improvements in their next call or conversation.

[0974] Specific examples

[0975] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[0976] 1. Audio Recording:

[0977] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[0978] 2. Sending audio data:

[0979] After the conversation ends, the device automatically sends the recording data to the server.

[0980] 3. Data analysis and feedback generation:

[0981] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data, performs sentiment analysis and keyword extraction, extracts points that need emphasis during the conversation, and generates feedback.

[0982] 4. Feedback received:

[0983] The device receives feedback from the server and provides specific advice such as "ask more open-ended questions to further draw out the customer's concerns."

[0984] 5. Implement improvements:

[0985] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[0986] In this way, users can continuously optimize their communication skills through data-driven coaching using AI. The system of the present invention can also significantly contribute to improving a company's business performance.

[0987] The processing flow will be explained below.

[0988] Step 1:

[0989] User:

[0990] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[0991] Step 2:

[0992] Device:

[0993] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[0994] Step 3:

[0995] Device:

[0996] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[0997] Step 4:

[0998] server:

[0999] The server receives the voice data transmitted from the terminal.

[1000] Step 5:

[1001] server:

[1002] The server performs noise reduction and volume normalization on the received audio data, improving the sound quality of the recorded data.

[1003] Step 6:

[1004] server:

[1005] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[1006] Step 7:

[1007] server:

[1008] The server uses natural language processing technology to perform sentiment analysis on the generated text data, classifying the emotional tone of the utterance as positive, negative, neutral, etc.

[1009] Step 8:

[1010] server:

[1011] The server simultaneously extracts keywords and important phrases from the text data, making the main points of the conversation clearer.

[1012] Step 9:

[1013] server:

[1014] The server evaluates user performance based on text data, sentiment analysis, and keyword extraction. Using an AI model trained on past data, it scores users based on the patterns of high and low performers.

[1015] Step 10:

[1016] server:

[1017] Based on the results of the performance evaluation, the server generates feedback to the user, including areas for improvement and specific advice.

[1018] Step 11:

[1019] server:

[1020] The server transmits the generated feedback to the user's terminal.

[1021] Step 12:

[1022] Device:

[1023] The terminal receives feedback from the server and visualizes it for the user to understand.

[1024] Step 13:

[1025] User:

[1026] The user can check the feedback displayed on the device and use it to make improvements in their next business call or sales conversation.

[1027] Example 1

[1028] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1029] Conventional systems for improving communication skills that use voice data have the problem of being unable to consistently perform advanced processing such as noise removal, volume normalization, emotion analysis, and keyword extraction when analyzing user voice data. Furthermore, the inability to provide real-time evaluation or feedback makes it difficult to quickly improve users' communication skills.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1031] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis and keyword extraction on the text data, means for evaluating performance based on the results of the sentiment analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for visualizing the feedback through a user interface. This makes it possible to perform advanced voice data preprocessing and natural language processing in real time and provide specific feedback to the user.

[1032] "Audio data" refers to data in which audio is recorded in digital format.

[1033] A "receiving means" is a device or program that has the function of receiving specific data from another device or system.

[1034] "Noise reduction" is the process of reducing or eliminating unwanted background noise in audio data.

[1035] "Volume normalization" is a process for making the volume level of audio data uniform.

[1036] "Means for converting to text data" refers to a system or algorithm for converting audio data into a corresponding string of characters.

[1037] "Sentiment analysis" is the process of analyzing and extracting the type and intensity of emotions from text data.

[1038] "Keyword extraction" is a technique for extracting important words and phrases from text data.

[1039] A "means for evaluating performance" is a device or program that has the function of evaluating a user's performance or results based on collected data.

[1040] "Feedback" refers to specific advice or suggestions provided to the user based on the evaluation results.

[1041] A "user interface" is the visual or operational means by which a user interacts with a system.

[1042] The present invention is a system that receives and analyzes voice data to improve a user's communication skills. This system is mainly composed of a "server," a "terminal," and a "user."

[1043] Hardware and Software Overview

[1044] server:

[1045] The server plays a central role in processing the audio data. The following software and tools are used for this processing:

[1046] Audio processing library: Librosa

[1047] Speech recognition engine: Google Cloud Speech-to-Text API

[1048] Natural language processing libraries: spaCy, NLTK

[1049] Machine learning libraries: Scikit-learn, TensorFlow

[1050] Device:

[1051] The terminal is a device that allows users to record audio and send and receive data to and from the server. Typical terminals are smartphones or tablets with built-in recording applications.

[1052] Processing flow and specific examples

[1053] Audio recording and transmission

[1054] The user starts the recording application on the device and records business calls or sales talks. The audio is temporarily stored on the device, and once recording is complete, it is encrypted and sent to the server.

[1055] Receiving and preprocessing audio data

[1056] The server receives the encrypted audio data, which is then denoised and normalised using the Librosa library, improving the quality of the audio data.

[1057] Conversion to text data

[1058] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data, which is then used for sentiment analysis and keyword extraction.

[1059] Sentiment analysis and keyword extraction

[1060] The server analyzes emotions from the transcribed text data and extracts important keywords. This process uses the spaCy and NLTK libraries. For example, it determines whether the emotion contained in the text data is "joy" or "anger."

[1061] Performance evaluation and feedback generation

[1062] The server evaluates the user's performance based on the results of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn and TensorFlow models trained on past data. Specific feedback is generated based on the evaluation results. For example, advice such as "You should ask more open-ended questions to further elicit customer concerns" is provided.

[1063] Send and visualize feedback

[1064] The server transmits the generated feedback to the user's terminal, which receives the feedback, visualizes it through a user interface, and presents it to the user.

[1065] Specific use cases

[1066] For example, the following scenario shows how a sales representative might use the system when approaching a new business deal.

[1067] 1. The sales representative (user) starts recording on the device while talking to the client.

[1068] 2. When the conversation ends, the device encrypts the recording and sends it to the server.

[1069] 3. The server receives the audio data and performs preprocessing, converting the audio into text data, and performing sentiment analysis and keyword extraction.

[1070] 4. The server generates feedback such as "You should simplify the explanations in areas where the customer is not interested."

[1071] 5. The device receives this feedback and presents it to the user in a visual form.

[1072] 6. The salesperson (user) uses this feedback in the next sales meeting to improve communication.

[1073] Prompt Sentence Examples

[1074] "Please analyze the audio data of this sales pitch and extract user sentiment and keywords. Please also include areas for improvement and specific advice in your feedback."

[1075] "Perform sentiment analysis on the following text data, extract keywords, and generate feedback: 'The customer asked many questions during the product explanation, which we answered well. However, you should ask more open-ended questions.'"

[1076] The above is a specific embodiment for carrying out the present invention. This system allows users to receive feedback in real time, enabling them to quickly improve their communication skills.

[1077] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1078] Step 1:

[1079] Audio recording and transmission

[1080] Device: The user launches the device's recording application and records a business call or sales conversation. Specifically, the device uses the built-in microphone to collect voice data (input) and temporarily stores it within the application. After the recording is complete, the voice data is encrypted. The encrypted voice data (output) is sent to the server using the HTTPS protocol.

[1081] Step 2:

[1082] Receiving audio data

[1083] Server: The server receives the encrypted voice data (input) sent from the device. Specifically, the server receives the data using the HTTPS protocol and stores it securely in a database. The stored voice data (output) is used for the next preprocessing step.

[1084] Step 3:

[1085] Audio data preprocessing

[1086] Server: The server performs noise reduction, volume normalization, and speaker separation on the stored audio data (input). This is done using the Librosa library. Specifically, the server analyzes the frequency of the audio signal and applies filters to remove noise. At the same time, it equalizes the audio volume level and separates multiple speakers if necessary. The preprocessed audio data (output) is used for the next transcription step.

[1087] Step 4:

[1088] Transcription of audio data

[1089] Server: The server converts the preprocessed audio data (input) into text data (output) using the Google Cloud Speech-to-Text API. Specifically, the server sends the audio data to the API and saves the returned transcription results in text format. This text data is used in the next steps of sentiment analysis and keyword extraction.

[1090] Step 5:

[1091] Sentiment analysis and keyword extraction

[1092] Server: The server performs sentiment analysis and keyword extraction on the transcribed text data (input). This is done using the spaCy and NLTK libraries. Specifically, it analyzes the text data and applies an algorithm to identify sentiment. At the same time, it processes and extracts important keywords. The analysis results (output) are used in the next performance evaluation step.

[1093] Step 6:

[1094] Performance Evaluation

[1095] Server: The server evaluates the user's performance based on the results (input) of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn or TensorFlow models that have learned from past data. Specifically, the server inputs data into the model and generates an evaluation score. The evaluation score (output) is used in the next feedback generation step.

[1096] Step 7:

[1097] Feedback Generation

[1098] Server: The server generates specific feedback based on the results of the performance evaluation (input). The generated feedback includes areas for improvement and specific advice. For example, the advice may be "You should provide more specific examples to attract customers' attention." The generated feedback (output) is provided to the user and is used in the next sending step.

[1099] Step 8:

[1100] Send and visualize feedback

[1101] Server: The server sends the generated feedback (input) to the user terminal. Specifically, the feedback data is encrypted again using the HTTPS protocol and then sent.

[1102] Terminal: The terminal receives the feedback (input) from the server and visualizes it through the user interface. The user can check the displayed feedback (output) and understand the specific improvements needed.

[1103] This is the specific flow of processing in the program for this system. In this way, advanced pre-processing of speech data and natural language processing make it possible to provide users with prompt and specific feedback.

[1104] (Application example 1)

[1105] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1106] In factory automation, the efficiency and accuracy of communication between workers and robots that give work instructions and status reports is important. However, with current systems, robots' responses to workers are often inappropriate, making it difficult to achieve natural, human-like dialogue. Therefore, in order to reduce worker stress and improve work efficiency, it is necessary to improve the robot's communication skills.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1108] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback and providing the feedback to the user, and means for analyzing and optimizing the voice data in real time so that the robot can appropriately provide work instructions and status reports. This improves the efficiency and accuracy of communication between the robot and the worker, reduces worker stress, and improves work efficiency.

[1109] "Voice data" refers to voice information collected from a user, and is the basis for analysis and processing.

[1110] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[1111] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[1112] "Text data" refers to data that has been converted from voice data into text format, and is subject to text analysis and sentiment analysis.

[1113] "Sentiment analysis" is the process of identifying emotions (e.g., joy, sadness, anger, etc.) from text data based on its expression and content.

[1114] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[1115] "Performance evaluation" is the process of evaluating the effectiveness and quality of a user's communication based on the results of sentiment analysis and keyword extraction.

[1116] "Feedback" refers to specific advice and information on areas for improvement that are provided to the user based on the evaluation results.

[1117] A "robot" is an automated machine that gives work instructions and reports on the status in factories and work sites.

[1118] "Real-time analytics" refers to the process of processing audio data instantly and making the analysis results immediately available.

[1119] "Optimization" is the process of improving a robot's communication skills and response methods to make them most effective.

[1120] The present invention is a system for optimizing interactions between robots and human workers in a factory environment, analyzing voice data in real time and generating feedback. Detailed description of the embodiments of the present invention is provided below.

[1121] System Overview

[1122] The system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends the voice data to the server. The server analyzes the voice data, generates appropriate feedback, and sends it to the terminal. The user receives the feedback and improves their communication skills based on it.

[1123] Hardware and Software

[1124] Audio recording and transmission

[1125] The device (smartphone, smart glasses, robot, etc.) is equipped with a microphone to record audio. The recorded data is encrypted for transmission to the server.

[1126] Audio data preprocessing

[1127] The server performs noise reduction and volume normalization of the audio data using the Python library "torchaudio."

[1128] Transcription of audio data

[1129] The server uses Facebook's Wav2Vec2 model to convert the audio data into text data.

[1130] Sentiment analysis and keyword extraction

[1131] For sentiment analysis and keyword extraction, we use the natural language processing library "Transformers," and also utilize models specialized for sentiment analysis.

[1132] Generate feedback

[1133] Leveraging generative AI models, it generates specific feedback based on sentiment analysis and keyword extraction, including specific improvements and recommendations.

[1134] Sending and Viewing Feedback

[1135] The feedback generated at the server is sent to the terminal and visualized through a user interface.

[1136] Specific examples

[1137] For example, consider a scenario in which a robot and a worker are interacting with each other to give new work instructions in a factory. During the work instructions, the robot records voice data in real time and sends it to a server.

[1138] The server analyzes the voice data, generates appropriate advice, and sends it back to the robot. For example, it may provide specific advice such as, "Speak more slowly so the worker can understand you." Through this process, the robot can gradually improve its communication with the worker.

[1139] Prompt Sentence Examples

[1140] You have designed a system to optimize interactions in a factory. It analyzes user voice data and provides feedback on improvements. What natural language processing model is used and how do you perform sentiment analysis?

[1141] In this way, the present invention provides a system that improves communication between robots and workers in factories, thereby increasing work efficiency, worker satisfaction, and overall business performance.

[1142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1143] Step 1: Record and send your audio

[1144] The device (smartphone or robot) uses a microphone to record audio data. The recorded audio data is encrypted and sent to a server. The input here is audio data, and the output is encrypted audio data. The specific operations are pressing the recording start button and sending the data after recording is complete.

[1145] Step 2: Receiving audio data

[1146] The server receives the encrypted voice data sent from the terminal. The input is the encrypted voice data, and the output is the received voice data. The specific operation is to execute the data reception protocol on the server side.

[1147] Step 3: Preprocessing the audio data

[1148] The server performs noise removal and volume normalization on the received audio data. The input here is the received audio data, and the output is the audio data that has been noise removed and volume normalized. Specifically, the data cleansing process uses the Python library "torchaudio."

[1149] Step 4: Transcribe the audio data

[1150] The server uses Facebook's Wav2Vec2 model to convert the audio data to text. The input is the cleansed audio data, and the output is the corresponding text data. The specific operation is to input the audio data into the model and obtain the resulting text.

[1151] Step 5: Sentiment analysis and keyword extraction

[1152] The server performs sentiment analysis on the text data and extracts important keywords. The software used is "transformers." The input is transcribed text data, and the output is sentiment classification data and extracted keywords. Specific operations include inputting text data into a natural language processing model and obtaining the analysis results.

[1153] Step 6: Performance evaluation

[1154] The server evaluates the user's communication performance based on the results of sentiment analysis and keyword extraction. The input is sentiment classification data and keywords, and the output is an evaluation score. Specific operations include running the evaluation algorithm and calculating the score.

[1155] Step 7: Generate feedback

[1156] The server generates specific feedback based on the results of the performance evaluation. It uses a generative AI model. The input is the evaluation score, and the output is a specific feedback message. The specific operation is to input the generative AI model and generate the resulting message.

[1157] Step 8: Submit and view feedback

[1158] The server sends the generated feedback to the device, which then displays it to the user. The input is the specific feedback message, and the output is the feedback display on the user interface. The specific operation is the message display process on the device's UI.

[1159] The above is a detailed description of the specific processing steps of the present invention, as well as the inputs and outputs and specific operations at each step.

[1160] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1161] The present invention is a system for improving communication skills by combining voice data and an emotion engine that recognizes the user's emotions. Below, the system program is explained in detail, and its processing is described with specific examples.

[1162] System configuration

[1163] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, recognizes the user's emotions using an emotion engine, evaluates the user's performance, and provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[1164] Program processing overview

[1165] server:

[1166] 1. Receiving audio data:

[1167] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[1168] 2. Preprocessing the audio data:

[1169] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[1170] 3. Transcription of audio data:

[1171] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[1172] 4. Sentiment analysis and keyword extraction:

[1173] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[1174] 5. Applying the Emotion Engine:

[1175] The server uses an emotion engine to analyze the user's emotional trends from both voice and text data, thereby recognizing the user's emotions in real time and reflecting them in the evaluation results.

[1176] 6. Performance Evaluation:

[1177] The server scores users' performance based on sentiment analysis and keyword extraction, using an AI model trained from past data to identify patterns of high and low performers.

[1178] 7. Feedback Generation:

[1179] The server generates feedback for the user based on the results of the performance evaluation, including suggestions for improvement and specific advice. Emotional trends based on the emotion engine are also added to the feedback.

[1180] 8. Submitting Feedback:

[1181] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[1182] Device:

[1183] 1. Audio recording and transmission:

[1184] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[1185] 2. Receiving Feedback:

[1186] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[1187] User:

[1188] 1. Start recording:

[1189] The user launches the application on the terminal and starts recording business calls or sales talks.

[1190] 2. Feedback check and implementation:

[1191] Users review the feedback they receive and implement improvements in their next call or conversation.

[1192] Specific examples

[1193] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[1194] 1. Audio Recording:

[1195] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[1196] 2. Sending audio data:

[1197] After the conversation ends, the device automatically sends the recording data to the server.

[1198] 3. Data Analysis:

[1199] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data and performs sentiment analysis and keyword extraction.

[1200] 4. Applying the Emotion Engine:

[1201] The server uses an emotion engine to analyze the user's emotional trends from the voice and text data, for example, identifying the user's feelings of tension or relief when answering a customer's question.

[1202] 5. Feedback Generation:

[1203] The server scores the user's performance and generates feedback based on the results of sentiment analysis, including specific advice such as, "You seemed a little nervous when answering the customer's question. Try to be more relaxed next time."

[1204] 6. Feedback received:

[1205] The terminal receives feedback from the server and displays it to the user.

[1206] 7. Implementing improvements:

[1207] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[1208] In this way, users can utilize the emotion engine to recognize and improve their own communication skills in more detail.The system of the present invention can also significantly contribute to improving the business performance of companies.

[1209] The processing flow will be explained below.

[1210] Step 1:

[1211] User:

[1212] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[1213] Step 2:

[1214] Device:

[1215] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[1216] Step 3:

[1217] Device:

[1218] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[1219] Step 4:

[1220] server:

[1221] The server receives the voice data transmitted from the terminal.

[1222] Step 5:

[1223] server:

[1224] The server performs noise reduction, volume normalization, and speaker separation on the received audio data, improving the quality of the recording.

[1225] Step 6:

[1226] server:

[1227] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[1228] Step 7:

[1229] server:

[1230] The server uses natural language processing technology on the generated text data to perform sentiment analysis and keyword extraction, thereby clarifying the emotional tone of the conversation and key phrases.

[1231] Step 8:

[1232] server:

[1233] The server uses an emotion engine to analyze the user's emotional trends in real time from voice and text data, specifically by evaluating changes in the user's tone of voice and speaking style to recognize their emotional state.

[1234] Step 9:

[1235] server:

[1236] The server scores users' performance based on the results of sentiment analysis, keyword extraction, and emotion engine analysis. Using an AI model that has learned from past data, scores are calculated based on the patterns of high and low performers.

[1237] Step 10:

[1238] server:

[1239] The server generates specific feedback based on the results of the performance evaluation, including analysis results including sentiment trends, and suggests areas for improvement and specific advice.

[1240] Step 11:

[1241] server:

[1242] The server transmits the generated feedback to the user's terminal.

[1243] Step 12:

[1244] Device:

[1245] The terminal receives feedback from the server and visualizes it for the user to understand.

[1246] Step 13:

[1247] User:

[1248] The user can check the feedback displayed on the device and try to improve it in the next business call or sales talk. The feedback also includes specific advice on how to change emotions.

[1249] Step 14:

[1250] User:

[1251] Users can also refer to past feedback, allowing them to optimize their communication skills through a process of incremental improvement.

[1252] Example 2

[1253] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1254] Conventional voice data analysis systems have struggled to accurately analyze users' emotions, evaluate their performance based on the results, and provide specific feedback. Furthermore, accuracy and real-time performance issues often exist in the processes of preprocessing voice data, converting it to text data, and analyzing emotions and generating feedback. This has hindered users' ability to effectively improve their communication skills.

[1255] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data from a user terminal, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing emotion analysis and keyword extraction on the text data using natural language processing technology, means for analyzing emotion trends from both the voice data and the text data, means for evaluating performance based on the results of the emotion analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for transmitting the feedback to the user terminal. This enables accurate pre-processing of voice data, emotion recognition, real-time analysis, and provision of specific feedback.

[1256] "User terminal" refers to a device used to record and play back audio data and to communicate data with a server.

[1257] "Voice data" refers to information that is a digital recording of the user's speech.

[1258] "Server" refers to a central processing unit that processes voice data received from a user terminal and generates analysis results and feedback.

[1259] "Noise reduction" refers to the process of removing background noise from audio data to improve the clarity of the audio.

[1260] "Volume normalization" refers to the process of adjusting the volume level of audio data to a constant level to make it easier to hear.

[1261] "Text data" refers to voice data converted into text information.

[1262] "Natural language processing technology" refers to technology for analyzing natural language text and extracting structured information.

[1263] "Sentiment analysis" refers to the process of identifying and evaluating a speaker's emotional state from text or audio data.

[1264] "Keyword extraction" refers to the process of extracting important words and phrases from text data.

[1265] "Emotional trends" refer to changes or trends in emotional states over a period of time.

[1266] "Performance evaluation" refers to the process of evaluating a user's response and attitude based on the results of sentiment analysis and keyword extraction.

[1267] "Feedback" refers to suggestions for improvement or advice provided to users based on the results of performance evaluation.

[1268] "Encryption" refers to the process of converting information using a specific algorithm to make it unreadable to third parties in order to send and receive data securely.

[1269] This invention is a system that improves a user's communication skills by combining voice data and an emotion engine that recognizes the user's emotions. This system is mainly composed of a "server," a "terminal," and a "user."

[1270] The server receives the audio data from the user's device, performs noise reduction and volume normalization using audio processing libraries such as Audacity and FFmpeg, and then converts the audio data into text using a speech recognition engine. Specifically, the server uses the Google Speech-to-Text API or IBM Watson Speech to Text.

[1271] For text data, natural language processing techniques such as TextBlob and NLTK are used to perform sentiment analysis and keyword extraction. An emotion engine is used to analyze both audio and text data and identify user sentiment trends in real time. This emotion engine may be implemented using a specific software module.

[1272] Furthermore, the server evaluates the user's performance based on the results of sentiment analysis and keyword extraction using a machine learning model (using TensorFlow or PyTorch) trained from past data. Based on the evaluation results, specific feedback is generated and sent to the user's device. This feedback also includes sentiment trends.

[1273] The device records the user's business calls and sales talks and sends the audio data to a server using a recording application such as Rev Recorder. The feedback data sent from the server is visually displayed on the device's user interface.

[1274] Users simply launch the application on their device, record their business calls or sales conversations, check the feedback displayed on the device, and implement improvements in their next call or conversation. This cycle allows users to continuously improve their communication skills.

[1275] As a concrete example, consider a scenario in which a sales representative is taking on a new business deal. The sales representative converses with the client while explaining the product. During the conversation, recording begins on the device, and after the conversation ends, the audio data is sent to the server. The server processes the audio data, performs sentiment analysis and performance evaluation, and then generates feedback. For example, specific advice such as "It's good to answer customer questions in a relaxed tone" is displayed on the user's device.

[1276] The advantage of this system is that it uses an emotion engine and machine learning models to help users improve their communication skills, creating a cycle of self-improvement that also contributes to improving the business performance of the company.

[1277] Example prompt sentence:

[1278] "I want to analyze voice data and recognize emotions. Please explain the specific steps of the system that processes this data and provides feedback to the user."

[1279] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1280] Step 1:

[1281] The server receives encrypted voice data from the user's device. This voice data is a digitally recorded file of the user's speech. After receiving it, the server decrypts the data using appropriate asymmetric encryption technology and stores it in a temporary folder for internal processing.

[1282] Step 2:

[1283] The server performs noise reduction on the received audio data. This process uses an audio processing library (e.g., Audacity or FFmpeg). It takes the decoded audio data as input and applies an algorithm to remove background noise to obtain clear audio output. It also performs volume normalization to ensure that all audio data has a consistent volume level.

[1284] Step 3:

[1285] The server inputs the noise-removed and volume-normalized audio data into a speech recognition engine, which converts it into text data. Specifically, it uses the Google Speech-to-Text or IBM Watson Speech to Text API. The speech recognition engine analyzes the input audio data, converts each syllable and word into text format, and outputs it.

[1286] Step 4:

[1287] The server analyzes the generated text data using natural language processing techniques. This process uses tools such as TextBlob and NLTK to extract emotional information and keywords from the text data. The input is the text data, and the output is an emotional label (positive, negative, neutral) and a list of key keywords.

[1288] Step 5:

[1289] The server uses an emotion engine to analyze emotional trends from both voice and text data. During this process, it analyzes factors such as tone, speed, and intensity of the voice to identify the user's emotional state. It uses text data and voice features as input data and obtains real-time emotional trend information as output.

[1290] Step 6:

[1291] The server evaluates user performance based on the sentiment analysis and keyword extraction results using a machine learning model (using TensorFlow or PyTorch). This machine learning model learns from past data and identifies patterns of high and low performers. It uses the sentiment analysis results and keyword list as input data and generates a performance evaluation score as output.

[1292] Step 7:

[1293] The server generates feedback for the user based on the performance evaluation results, including specific improvements and advice. This feedback also includes the emotional trend obtained by the emotional engine. The server uses the performance evaluation score and the emotional trend information as inputs and generates a feedback message as output.

[1294] Step 8:

[1295] The server sends the generated feedback to the user's device. This data is then encrypted again and sent securely to the user's device. The device receives the feedback data and visually displays it in the user interface. The user can review this feedback and implement improvements in their next business call or sales conversation.

[1296] (Application example 2)

[1297] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1298] In modern factories, cooperation between workers and machines is extremely important. However, when workers are under high stress or tension, communication efficiency declines, which can have a negative impact on productivity and safety. Current technology lacks a mechanism for recognizing workers' emotional state in real time and providing immediate, specific feedback. Therefore, there is a need for a system that can manage workers' stress and tension, maintain smooth communication, and improve work efficiency.

[1299] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1300] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis on the text data, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for supporting communication between factory workers and machines in real time and providing feedback based on the worker's emotional state. This makes it possible to grasp the worker's emotional state in real time and provide prompt and appropriate feedback.

[1301] "Audio data" refers to data that represents audio in digital form.

[1302] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[1303] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[1304] "Text data" refers to data obtained by converting voice data into characters.

[1305] "Sentiment analysis" is a technology that identifies emotional states from text and audio data.

[1306] "Keyword extraction" is the process of extracting important words and phrases from text data.

[1307] "Performance evaluation" refers to evaluating a user's behavior and abilities based on the results of sentiment analysis and keyword extraction.

[1308] "Feedback" refers to suggestions for improvement and advice provided to users based on the evaluation results.

[1309] "Workers" refers to personnel who perform work within a factory.

[1310] "Machinery" refers to various devices and robots used within the factory.

[1311] "Real-time communication support" means instantly analyzing and supporting the dialogue between workers and machines.

[1312] "Emotional state" refers to the psychological state of a worker, such as stress or tension.

[1313] "Operation guidance" refers to instructing workers on appropriate work methods and countermeasures based on their emotional state.

[1314] The system of this invention supports real-time communication between factory workers and machines, improving work efficiency and safety. The system has three main components: a server, a terminal, and a user.

[1315] Server Functionality Description

[1316] The server is responsible for processing the received audio data. The specific processing flow is as follows:

[1317] 1. Receive audio data:

[1318] The server receives the voice data sent from the device. The received data is securely encrypted.

[1319] 2. Preprocessing the audio data:

[1320] The server performs noise reduction and volume normalization on the received audio data using pydub.

[1321] 3. Transcription of audio data:

[1322] The server uses a speech recognition model (such as Huggingface's Wav2Vec2) to convert the audio data into text.

[1323] 4. Sentiment analysis and keyword extraction:

[1324] The server then uses natural language processing technology and a sentiment analysis engine to analyze sentiment and extract keywords from the converted text data. Sentiment analysis is performed using the Huggingface pipeline.

[1325] 5. Performance evaluation and feedback generation:

[1326] The server evaluates the worker's performance based on the results of emotion analysis and keyword extraction, and generates specific feedback and operational guidance according to the worker's emotional state.

[1327] Device function description

[1328] A terminal is a device that users use in a factory. Its specific operations are as follows:

[1329] 1. Audio recording and transmission:

[1330] The device records the worker's voice in real time and sends the recorded data to a server, where it is temporarily stored and encrypted.

[1331] 2. Receiving and Viewing Feedback:

[1332] The device receives the feedback sent from the server and displays it to the user in an easy-to-understand manner, including operational guidance based on the user's emotional state.

[1333] User behavior description

[1334] Users are primarily factory workers who use the system to improve their own work. Specific examples of actions include:

[1335] 1. Start recording:

[1336] The worker starts recording the voice using the terminal at the same time as starting the work.

[1337] 2. Use feedback:

[1338] After receiving the feedback, the worker follows the instructions and implements the improvements in the next work process. For example, if tension is detected during work, the worker follows specific advice such as "take a deep breath and relax."

[1339] Examples and prompts

[1340] As a concrete example, consider the following scenario: a factory worker issues a warning about a malfunctioning machine:

[1341] Worker voice recording: "There is an abnormal noise in the machine's bearings, please check for oil leaks on the right side."

[1342] Sentiment analysis results: Detected that the worker was nervous.

[1343] Feedback: "We are in a tense situation, so please refer to the emergency response checklist for when an emergency occurs."

[1344] An example of a prompt for a generative AI model: "There is an abnormal noise in the machine's bearings, so please check the right side for oil leaks."

[1345] In this way, the system of the present invention can recognize the emotional state of workers in real time and provide appropriate feedback, thereby improving the quality of communication within the factory and significantly improving work efficiency and safety.

[1346] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1347] Step 1:

[1348] Audio recording and transmission

[1349] Description: The device records the worker's voice in real time and sends the recorded data to the server.

[1350] Input: Worker's voice input.

[1351] Output: Encrypted audio data.

[1352] How it works: The device records the worker's voice in real time and temporarily stores the data. The stored data is encrypted and sent to a server using a secure protocol (e.g., HTTPS).

[1353] Step 2:

[1354] Audio data preprocessing

[1355] Description: The server performs noise reduction and volume normalization on the received audio data.

[1356] Input: Encrypted audio data received from the device.

[1357] Output: Denoised and volume-normalised audio data.

[1358] Specific operation: The server decrypts the received audio data based on the encryption method, and uses the pydub library to perform noise removal and volume normalization.

[1359] Step 3:

[1360] Transcription of audio data

[1361] Description: The server uses a speech recognition model to convert the audio data into text data.

[1362] Input: Denoised and volume-normalised audio data.

[1363] Output: Text data.

[1364] Specific operation: The server inputs the noise-removed and volume-normalized audio data into a speech recognition model (e.g., Huggingface's Wav2Vec2) and converts the audio into corresponding text data.

[1365] Step 4:

[1366] Sentiment analysis and keyword extraction

[1367] Description: The server performs sentiment analysis and keyword extraction on the converted text data.

[1368] Input: Transcribed text data.

[1369] Output: Emotional state and keyword extraction results.

[1370] Specific operation: The server inputs text data into the Huggingface pipeline and performs sentiment analysis and keyword extraction. The obtained emotional state and keywords are used to evaluate performance in the next step.

[1371] Step 5:

[1372] Performance evaluation and feedback generation

[1373] Description: The server evaluates the worker's performance based on the results of sentiment analysis and keyword extraction, and generates specific feedback and operation guidance.

[1374] Input: Emotional state and keyword extraction results.

[1375] Output: Specific feedback and operation guidance.

[1376] Specific Action: The server integrates the results of sentiment analysis and keyword extraction to evaluate the worker's overall performance, including stress and tension. The generated feedback includes specific improvements and operational guidance.

[1377] Step 6:

[1378] Receiving and viewing feedback

[1379] Description: The device receives feedback sent from the server and displays it visually to the user.

[1380] Input: Feedback received from the server.

[1381] Output: Feedback information that is displayed to the user.

[1382] Specific operation: The device asynchronously receives feedback sent from the server and displays it in an easy-to-understand visualization format for the user, allowing the user to check work improvements and emergency response measures in real time.

[1383] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1384] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1385] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1386] [Fourth embodiment]

[1387] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1388] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1389] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1390] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1391] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1392] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1393] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1394] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1395] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1396] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1397] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1398] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1399] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1400] The present invention is a system that receives voice data and analyzes it to improve a user's communication skills. The system program will be explained below, and its processing will be described with specific examples.

[1401] System Overview

[1402] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, evaluates the performance, and then provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[1403] Program processing overview

[1404] server:

[1405] 1. Receiving audio data:

[1406] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[1407] 2. Preprocessing the audio data:

[1408] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[1409] 3. Transcription of audio data:

[1410] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[1411] 4. Sentiment analysis and keyword extraction:

[1412] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[1413] 5. Performance Evaluation:

[1414] The server scores users' performance based on sentiment analysis and keyword extraction. The AI ​​model learns from past data and identifies patterns of high and low performers.

[1415] 6. Feedback Generation:

[1416] The server generates specific feedback based on the results of the performance evaluation, including areas for improvement and specific advice.

[1417] 7. Submitting Feedback:

[1418] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[1419] Device:

[1420] 1. Audio recording and transmission:

[1421] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[1422] 2. Receiving Feedback:

[1423] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[1424] User:

[1425] 1. Start recording:

[1426] The user launches the application on the terminal and starts recording business calls or sales talks.

[1427] 2. Feedback check and implementation:

[1428] Users review the feedback they receive and implement improvements in their next call or conversation.

[1429] Specific examples

[1430] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[1431] 1. Audio Recording:

[1432] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[1433] 2. Sending audio data:

[1434] After the conversation ends, the device automatically sends the recording data to the server.

[1435] 3. Data analysis and feedback generation:

[1436] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data, performs sentiment analysis and keyword extraction, extracts points that need emphasis during the conversation, and generates feedback.

[1437] 4. Feedback received:

[1438] The device receives feedback from the server and provides specific advice such as "ask more open-ended questions to further draw out the customer's concerns."

[1439] 5. Implement improvements:

[1440] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[1441] In this way, users can continuously optimize their communication skills through data-driven coaching using AI. The system of the present invention can also significantly contribute to improving a company's business performance.

[1442] The processing flow will be explained below.

[1443] Step 1:

[1444] User:

[1445] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[1446] Step 2:

[1447] Device:

[1448] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[1449] Step 3:

[1450] Device:

[1451] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[1452] Step 4:

[1453] server:

[1454] The server receives the voice data transmitted from the terminal.

[1455] Step 5:

[1456] server:

[1457] The server performs noise reduction and volume normalization on the received audio data, improving the sound quality of the recorded data.

[1458] Step 6:

[1459] server:

[1460] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[1461] Step 7:

[1462] server:

[1463] The server uses natural language processing technology to perform sentiment analysis on the generated text data, classifying the emotional tone of the utterance as positive, negative, neutral, etc.

[1464] Step 8:

[1465] server:

[1466] The server simultaneously extracts keywords and important phrases from the text data, making the main points of the conversation clearer.

[1467] Step 9:

[1468] server:

[1469] The server evaluates user performance based on text data, sentiment analysis, and keyword extraction. Using an AI model trained on past data, it scores users based on the patterns of high and low performers.

[1470] Step 10:

[1471] server:

[1472] Based on the results of the performance evaluation, the server generates feedback to the user, including areas for improvement and specific advice.

[1473] Step 11:

[1474] server:

[1475] The server transmits the generated feedback to the user's terminal.

[1476] Step 12:

[1477] Device:

[1478] The terminal receives feedback from the server and visualizes it for the user to understand.

[1479] Step 13:

[1480] User:

[1481] The user can check the feedback displayed on the device and use it to make improvements in their next business call or sales conversation.

[1482] Example 1

[1483] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1484] Conventional systems for improving communication skills that use voice data have the problem of being unable to consistently perform advanced processing such as noise removal, volume normalization, emotion analysis, and keyword extraction when analyzing user voice data. Furthermore, the inability to provide real-time evaluation or feedback makes it difficult to quickly improve users' communication skills.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1486] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis and keyword extraction on the text data, means for evaluating performance based on the results of the sentiment analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for visualizing the feedback through a user interface. This makes it possible to perform advanced voice data preprocessing and natural language processing in real time and provide specific feedback to the user.

[1487] "Audio data" refers to data in which audio is recorded in digital format.

[1488] A "receiving means" is a device or program that has the function of receiving specific data from another device or system.

[1489] "Noise reduction" is the process of reducing or eliminating unwanted background noise in audio data.

[1490] "Volume normalization" is a process for making the volume level of audio data uniform.

[1491] "Means for converting to text data" refers to a system or algorithm for converting audio data into a corresponding string of characters.

[1492] "Sentiment analysis" is the process of analyzing and extracting the type and intensity of emotions from text data.

[1493] "Keyword extraction" is a technique for extracting important words and phrases from text data.

[1494] A "means for evaluating performance" is a device or program that has the function of evaluating a user's performance or results based on collected data.

[1495] "Feedback" refers to specific advice or suggestions provided to the user based on the evaluation results.

[1496] A "user interface" is the visual or operational means by which a user interacts with a system.

[1497] The present invention is a system that receives and analyzes voice data to improve a user's communication skills. This system is mainly composed of a "server," a "terminal," and a "user."

[1498] Hardware and Software Overview

[1499] server:

[1500] The server plays a central role in processing the audio data. The following software and tools are used for this processing:

[1501] Audio processing library: Librosa

[1502] Speech recognition engine: Google Cloud Speech-to-Text API

[1503] Natural language processing libraries: spaCy, NLTK

[1504] Machine learning libraries: Scikit-learn, TensorFlow

[1505] Device:

[1506] The terminal is a device that allows users to record audio and send and receive data to and from the server. Typical terminals are smartphones or tablets with built-in recording applications.

[1507] Processing flow and specific examples

[1508] Audio recording and transmission

[1509] The user starts the recording application on the device and records business calls or sales talks. The audio is temporarily stored on the device, and once recording is complete, it is encrypted and sent to the server.

[1510] Receiving and preprocessing audio data

[1511] The server receives the encrypted audio data, which is then denoised and normalised using the Librosa library, improving the quality of the audio data.

[1512] Conversion to text data

[1513] The server uses the Google Cloud Speech-to-Text API to convert the audio data into text data, which is then used for sentiment analysis and keyword extraction.

[1514] Sentiment analysis and keyword extraction

[1515] The server analyzes emotions from the transcribed text data and extracts important keywords. This process uses the spaCy and NLTK libraries. For example, it determines whether the emotion contained in the text data is "joy" or "anger."

[1516] Performance evaluation and feedback generation

[1517] The server evaluates the user's performance based on the results of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn and TensorFlow models trained on past data. Specific feedback is generated based on the evaluation results. For example, advice such as "You should ask more open-ended questions to further elicit customer concerns" is provided.

[1518] Send and visualize feedback

[1519] The server transmits the generated feedback to the user's terminal, which receives the feedback, visualizes it through a user interface, and presents it to the user.

[1520] Specific use cases

[1521] For example, the following scenario shows how a sales representative might use the system when approaching a new business deal.

[1522] 1. The sales representative (user) starts recording on the device while talking to the client.

[1523] 2. When the conversation ends, the device encrypts the recording and sends it to the server.

[1524] 3. The server receives the audio data and performs preprocessing, converting the audio into text data, and performing sentiment analysis and keyword extraction.

[1525] 4. The server generates feedback such as "You should simplify the explanations in areas where the customer is not interested."

[1526] 5. The device receives this feedback and presents it to the user in a visual form.

[1527] 6. The salesperson (user) uses this feedback in the next sales meeting to improve communication.

[1528] Prompt Sentence Examples

[1529] "Please analyze the audio data of this sales pitch and extract user sentiment and keywords. Please also include areas for improvement and specific advice in your feedback."

[1530] "Perform sentiment analysis on the following text data, extract keywords, and generate feedback: 'The customer asked many questions during the product explanation, which we answered well. However, you should ask more open-ended questions.'"

[1531] The above is a specific embodiment for carrying out the present invention. This system allows users to receive feedback in real time, enabling them to quickly improve their communication skills.

[1532] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1533] Step 1:

[1534] Audio recording and transmission

[1535] Device: The user launches the device's recording application and records a business call or sales conversation. Specifically, the device uses the built-in microphone to collect voice data (input) and temporarily stores it within the application. After the recording is complete, the voice data is encrypted. The encrypted voice data (output) is sent to the server using the HTTPS protocol.

[1536] Step 2:

[1537] Receiving audio data

[1538] Server: The server receives the encrypted voice data (input) sent from the device. Specifically, the server receives the data using the HTTPS protocol and stores it securely in a database. The stored voice data (output) is used for the next preprocessing step.

[1539] Step 3:

[1540] Audio data preprocessing

[1541] Server: The server performs noise reduction, volume normalization, and speaker separation on the stored audio data (input). This is done using the Librosa library. Specifically, the server analyzes the frequency of the audio signal and applies filters to remove noise. At the same time, it equalizes the audio volume level and separates multiple speakers if necessary. The preprocessed audio data (output) is used for the next transcription step.

[1542] Step 4:

[1543] Transcription of audio data

[1544] Server: The server converts the preprocessed audio data (input) into text data (output) using the Google Cloud Speech-to-Text API. Specifically, the server sends the audio data to the API and saves the returned transcription results in text format. This text data is used in the next steps of sentiment analysis and keyword extraction.

[1545] Step 5:

[1546] Sentiment analysis and keyword extraction

[1547] Server: The server performs sentiment analysis and keyword extraction on the transcribed text data (input). This is done using the spaCy and NLTK libraries. Specifically, it analyzes the text data and applies an algorithm to identify sentiment. At the same time, it processes and extracts important keywords. The analysis results (output) are used in the next performance evaluation step.

[1548] Step 6:

[1549] Performance Evaluation

[1550] Server: The server evaluates the user's performance based on the results (input) of sentiment analysis and keyword extraction. This evaluation uses Scikit-learn or TensorFlow models that have learned from past data. Specifically, the server inputs data into the model and generates an evaluation score. The evaluation score (output) is used in the next feedback generation step.

[1551] Step 7:

[1552] Feedback Generation

[1553] Server: The server generates specific feedback based on the results of the performance evaluation (input). The generated feedback includes areas for improvement and specific advice. For example, the advice may be "You should provide more specific examples to attract customers' attention." The generated feedback (output) is provided to the user and is used in the next sending step.

[1554] Step 8:

[1555] Send and visualize feedback

[1556] Server: The server sends the generated feedback (input) to the user terminal. Specifically, the feedback data is encrypted again using the HTTPS protocol and then sent.

[1557] Terminal: The terminal receives the feedback (input) from the server and visualizes it through the user interface. The user can check the displayed feedback (output) and understand the specific improvements needed.

[1558] This is the specific flow of processing in the program for this system. In this way, advanced pre-processing of speech data and natural language processing make it possible to provide users with prompt and specific feedback.

[1559] (Application example 1)

[1560] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1561] In factory automation, the efficiency and accuracy of communication between workers and robots that give work instructions and status reports is important. However, with current systems, robots' responses to workers are often inappropriate, making it difficult to achieve natural, human-like dialogue. Therefore, in order to reduce worker stress and improve work efficiency, it is necessary to improve the robot's communication skills.

[1562] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1563] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback and providing the feedback to the user, and means for analyzing and optimizing the voice data in real time so that the robot can appropriately provide work instructions and status reports. This improves the efficiency and accuracy of communication between the robot and the worker, reduces worker stress, and improves work efficiency.

[1564] "Voice data" refers to voice information collected from a user, and is the basis for analysis and processing.

[1565] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[1566] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[1567] "Text data" refers to data that has been converted from voice data into text format, and is subject to text analysis and sentiment analysis.

[1568] "Sentiment analysis" is the process of identifying emotions (e.g., joy, sadness, anger, etc.) from text data based on its expression and content.

[1569] "Keyword extraction" is the process of identifying and extracting important words and phrases from text data.

[1570] "Performance evaluation" is the process of evaluating the effectiveness and quality of a user's communication based on the results of sentiment analysis and keyword extraction.

[1571] "Feedback" refers to specific advice and information on areas for improvement that are provided to the user based on the evaluation results.

[1572] A "robot" is an automated machine that gives work instructions and reports on the status in factories and work sites.

[1573] "Real-time analytics" refers to the process of processing audio data instantly and making the analysis results immediately available.

[1574] "Optimization" is the process of improving a robot's communication skills and response methods to make them most effective.

[1575] The present invention is a system for optimizing interactions between robots and human workers in a factory environment, analyzing voice data in real time and generating feedback. Detailed description of the embodiments of the present invention is provided below.

[1576] System Overview

[1577] The system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends the voice data to the server. The server analyzes the voice data, generates appropriate feedback, and sends it to the terminal. The user receives the feedback and improves their communication skills based on it.

[1578] Hardware and Software

[1579] Audio recording and transmission

[1580] The device (smartphone, smart glasses, robot, etc.) is equipped with a microphone to record audio. The recorded data is encrypted for transmission to the server.

[1581] Audio data preprocessing

[1582] The server performs noise reduction and volume normalization of the audio data using the Python library "torchaudio."

[1583] Transcription of audio data

[1584] The server uses Facebook's Wav2Vec2 model to convert the audio data into text data.

[1585] Sentiment analysis and keyword extraction

[1586] For sentiment analysis and keyword extraction, we use the natural language processing library "Transformers," and also utilize models specialized for sentiment analysis.

[1587] Generate feedback

[1588] Leveraging generative AI models, it generates specific feedback based on sentiment analysis and keyword extraction, including specific improvements and recommendations.

[1589] Sending and Viewing Feedback

[1590] The feedback generated at the server is sent to the terminal and visualized through a user interface.

[1591] Specific examples

[1592] For example, consider a scenario in which a robot and a worker are interacting with each other to give new work instructions in a factory. During the work instructions, the robot records voice data in real time and sends it to a server.

[1593] The server analyzes the voice data, generates appropriate advice, and sends it back to the robot. For example, it may provide specific advice such as, "Speak more slowly so the worker can understand you." Through this process, the robot can gradually improve its communication with the worker.

[1594] Prompt Sentence Examples

[1595] You have designed a system to optimize interactions in a factory. It analyzes user voice data and provides feedback on improvements. What natural language processing model is used and how do you perform sentiment analysis?

[1596] In this way, the present invention provides a system that improves communication between robots and workers in factories, thereby increasing work efficiency, worker satisfaction, and overall business performance.

[1597] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1598] Step 1: Record and send your audio

[1599] The device (smartphone or robot) uses a microphone to record audio data. The recorded audio data is encrypted and sent to a server. The input here is audio data, and the output is encrypted audio data. The specific operations are pressing the recording start button and sending the data after recording is complete.

[1600] Step 2: Receiving audio data

[1601] The server receives the encrypted voice data sent from the terminal. The input is the encrypted voice data, and the output is the received voice data. The specific operation is to execute the data reception protocol on the server side.

[1602] Step 3: Preprocessing the audio data

[1603] The server performs noise removal and volume normalization on the received audio data. The input here is the received audio data, and the output is the audio data that has been noise removed and volume normalized. Specifically, the data cleansing process uses the Python library "torchaudio."

[1604] Step 4: Transcribe the audio data

[1605] The server uses Facebook's Wav2Vec2 model to convert the audio data to text. The input is the cleansed audio data, and the output is the corresponding text data. The specific operation is to input the audio data into the model and obtain the resulting text.

[1606] Step 5: Sentiment analysis and keyword extraction

[1607] The server performs sentiment analysis on the text data and extracts important keywords. The software used is "transformers." The input is transcribed text data, and the output is sentiment classification data and extracted keywords. Specific operations include inputting text data into a natural language processing model and obtaining the analysis results.

[1608] Step 6: Performance evaluation

[1609] The server evaluates the user's communication performance based on the results of sentiment analysis and keyword extraction. The input is sentiment classification data and keywords, and the output is an evaluation score. Specific operations include running the evaluation algorithm and calculating the score.

[1610] Step 7: Generate feedback

[1611] The server generates specific feedback based on the results of the performance evaluation. It uses a generative AI model. The input is the evaluation score, and the output is a specific feedback message. The specific operation is to input the generative AI model and generate the resulting message.

[1612] Step 8: Submit and view feedback

[1613] The server sends the generated feedback to the device, which then displays it to the user. The input is the specific feedback message, and the output is the feedback display on the user interface. The specific operation is the message display process on the device's UI.

[1614] The above is a detailed description of the specific processing steps of the present invention, as well as the inputs and outputs and specific operations at each step.

[1615] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1616] The present invention is a system for improving communication skills by combining voice data and an emotion engine that recognizes the user's emotions. Below, the system program is explained in detail, and its processing is described with specific examples.

[1617] System configuration

[1618] This system mainly consists of a "server," a "terminal," and a "user." The terminal records the user's voice and sends it to the server. The server analyzes the voice data, recognizes the user's emotions using an emotion engine, evaluates the user's performance, and provides feedback to the user. The user receives the feedback and optimizes their communication skills.

[1619] Program processing overview

[1620] server:

[1621] 1. Receiving audio data:

[1622] The server receives the voice data from the user terminal, which is securely encrypted and then transmitted.

[1623] 2. Preprocessing the audio data:

[1624] The server performs noise reduction, volume normalization, and speaker separation on the received audio data.

[1625] 3. Transcription of audio data:

[1626] The server uses a speech recognition engine to convert the voice data into corresponding text data.

[1627] 4. Sentiment analysis and keyword extraction:

[1628] The server uses natural language processing technology on the text data to perform sentiment analysis and keyword extraction.

[1629] 5. Applying the Emotion Engine:

[1630] The server uses an emotion engine to analyze the user's emotional trends from both voice and text data, thereby recognizing the user's emotions in real time and reflecting them in the evaluation results.

[1631] 6. Performance Evaluation:

[1632] The server scores users' performance based on sentiment analysis and keyword extraction, using an AI model trained from past data to identify patterns of high and low performers.

[1633] 7. Feedback Generation:

[1634] The server generates feedback for the user based on the results of the performance evaluation, including suggestions for improvement and specific advice. Emotional trends based on the emotion engine are also added to the feedback.

[1635] 8. Submitting Feedback:

[1636] The server transmits the generated feedback to the user terminal and displays it on the user interface.

[1637] Device:

[1638] 1. Audio recording and transmission:

[1639] The device records business calls and sales talks made by the user, and the recorded data is sent to the server.

[1640] 2. Receiving Feedback:

[1641] The device receives the feedback sent from the server, visualizes it, and presents it to the user.

[1642] User:

[1643] 1. Start recording:

[1644] The user launches the application on the terminal and starts recording business calls or sales talks.

[1645] 2. Feedback check and implementation:

[1646] Users review the feedback they receive and implement improvements in their next call or conversation.

[1647] Specific examples

[1648] For example, consider a specific scenario in which a sales representative is approaching a new business deal.

[1649] 1. Audio Recording:

[1650] The salesperson (user) talks with the client while explaining the product. During the conversation, the terminal starts recording.

[1651] 2. Sending audio data:

[1652] After the conversation ends, the device automatically sends the recording data to the server.

[1653] 3. Data Analysis:

[1654] The server receives the recorded data and performs preprocessing (noise removal, volume normalization), then converts the audio data into text data and performs sentiment analysis and keyword extraction.

[1655] 4. Applying the Emotion Engine:

[1656] The server uses an emotion engine to analyze the user's emotional trends from the voice and text data, for example, identifying the user's feelings of tension or relief when answering a customer's question.

[1657] 5. Feedback Generation:

[1658] The server scores the user's performance and generates feedback based on the results of sentiment analysis, including specific advice such as, "You seemed a little nervous when answering the customer's question. Try to be more relaxed next time."

[1659] 6. Feedback received:

[1660] The terminal receives feedback from the server and displays it to the user.

[1661] 7. Implementing improvements:

[1662] The sales representative (user) will use the feedback in their next sales meeting to improve the content of their talk and strengthen their dialogue with the customer.

[1663] In this way, users can utilize the emotion engine to recognize and improve their own communication skills in more detail.The system of the present invention can also significantly contribute to improving the business performance of companies.

[1664] The processing flow will be explained below.

[1665] Step 1:

[1666] User:

[1667] The user launches the application on the terminal and presses the record button to start recording a business call or sales pitch.

[1668] Step 2:

[1669] Device:

[1670] The device records the user's voice in real time and temporarily stores the voice data at regular intervals according to rules.

[1671] Step 3:

[1672] Device:

[1673] Once the conversation is over, the device encrypts the recorded audio data and sends it securely to a designated server.

[1674] Step 4:

[1675] server:

[1676] The server receives the voice data transmitted from the terminal.

[1677] Step 5:

[1678] server:

[1679] The server performs noise reduction, volume normalization, and speaker separation on the received audio data, improving the quality of the recording.

[1680] Step 6:

[1681] server:

[1682] The server transcribes the preprocessed audio data into text data, and then uses a speech recognition engine to convert the audio data into text.

[1683] Step 7:

[1684] server:

[1685] The server uses natural language processing technology on the generated text data to perform sentiment analysis and keyword extraction, thereby clarifying the emotional tone of the conversation and key phrases.

[1686] Step 8:

[1687] server:

[1688] The server uses an emotion engine to analyze the user's emotional trends in real time from voice and text data, specifically by evaluating changes in the user's tone of voice and speaking style to recognize their emotional state.

[1689] Step 9:

[1690] server:

[1691] The server scores users' performance based on the results of sentiment analysis, keyword extraction, and emotion engine analysis. Using an AI model that has learned from past data, scores are calculated based on the patterns of high and low performers.

[1692] Step 10:

[1693] server:

[1694] The server generates specific feedback based on the results of the performance evaluation, including analysis results including sentiment trends, and suggests areas for improvement and specific advice.

[1695] Step 11:

[1696] server:

[1697] The server transmits the generated feedback to the user's terminal.

[1698] Step 12:

[1699] Device:

[1700] The terminal receives feedback from the server and visualizes it for the user to understand.

[1701] Step 13:

[1702] User:

[1703] The user can check the feedback displayed on the device and try to improve it in the next business call or sales talk. The feedback also includes specific advice on how to change emotions.

[1704] Step 14:

[1705] User:

[1706] Users can also refer to past feedback, allowing them to optimize their communication skills through a process of incremental improvement.

[1707] Example 2

[1708] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1709] Conventional voice data analysis systems have struggled to accurately analyze users' emotions, evaluate their performance based on the results, and provide specific feedback. Furthermore, accuracy and real-time performance issues often exist in the processes of preprocessing voice data, converting it to text data, and analyzing emotions and generating feedback. This has hindered users' ability to effectively improve their communication skills.

[1710] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving voice data from a user terminal, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing emotion analysis and keyword extraction on the text data using natural language processing technology, means for analyzing emotion trends from both the voice data and the text data, means for evaluating performance based on the results of the emotion analysis and keyword extraction, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for transmitting the feedback to the user terminal. This enables accurate pre-processing of voice data, emotion recognition, real-time analysis, and provision of specific feedback.

[1711] "User terminal" refers to a device used to record and play back audio data and to communicate data with a server.

[1712] "Voice data" refers to information that is a digital recording of the user's speech.

[1713] "Server" refers to a central processing unit that processes voice data received from a user terminal and generates analysis results and feedback.

[1714] "Noise reduction" refers to the process of removing background noise from audio data to improve the clarity of the audio.

[1715] "Volume normalization" refers to the process of adjusting the volume level of audio data to a constant level to make it easier to hear.

[1716] "Text data" refers to voice data converted into text information.

[1717] "Natural language processing technology" refers to technology for analyzing natural language text and extracting structured information.

[1718] "Sentiment analysis" refers to the process of identifying and evaluating a speaker's emotional state from text or audio data.

[1719] "Keyword extraction" refers to the process of extracting important words and phrases from text data.

[1720] "Emotional trends" refer to changes or trends in emotional states over a period of time.

[1721] "Performance evaluation" refers to the process of evaluating a user's response and attitude based on the results of sentiment analysis and keyword extraction.

[1722] "Feedback" refers to suggestions for improvement or advice provided to users based on the results of performance evaluation.

[1723] "Encryption" refers to the process of converting information using a specific algorithm to make it unreadable to third parties in order to send and receive data securely.

[1724] This invention is a system that improves a user's communication skills by combining voice data and an emotion engine that recognizes the user's emotions. This system is mainly composed of a "server," a "terminal," and a "user."

[1725] The server receives the audio data from the user's device, performs noise reduction and volume normalization using audio processing libraries such as Audacity and FFmpeg, and then converts the audio data into text using a speech recognition engine. Specifically, the server uses the Google Speech-to-Text API or IBM Watson Speech to Text.

[1726] For text data, natural language processing techniques such as TextBlob and NLTK are used to perform sentiment analysis and keyword extraction. An emotion engine is used to analyze both audio and text data and identify user sentiment trends in real time. This emotion engine may be implemented using a specific software module.

[1727] Furthermore, the server evaluates the user's performance based on the results of sentiment analysis and keyword extraction using a machine learning model (using TensorFlow or PyTorch) trained from past data. Based on the evaluation results, specific feedback is generated and sent to the user's device. This feedback also includes sentiment trends.

[1728] The device records the user's business calls and sales talks and sends the audio data to a server using a recording application such as Rev Recorder. The feedback data sent from the server is visually displayed on the device's user interface.

[1729] Users simply launch the application on their device, record their business calls or sales conversations, check the feedback displayed on the device, and implement improvements in their next call or conversation. This cycle allows users to continuously improve their communication skills.

[1730] As a concrete example, consider a scenario in which a sales representative is taking on a new business deal. The sales representative converses with the client while explaining the product. During the conversation, recording begins on the device, and after the conversation ends, the audio data is sent to the server. The server processes the audio data, performs sentiment analysis and performance evaluation, and then generates feedback. For example, specific advice such as "It's good to answer customer questions in a relaxed tone" is displayed on the user's device.

[1731] The advantage of this system is that it uses an emotion engine and machine learning models to help users improve their communication skills, creating a cycle of self-improvement that also contributes to improving the business performance of the company.

[1732] Example prompt sentence:

[1733] "I want to analyze voice data and recognize emotions. Please explain the specific steps of the system that processes this data and provides feedback to the user."

[1734] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1735] Step 1:

[1736] The server receives encrypted voice data from the user's device. This voice data is a digitally recorded file of the user's speech. After receiving it, the server decrypts the data using appropriate asymmetric encryption technology and stores it in a temporary folder for internal processing.

[1737] Step 2:

[1738] The server performs noise reduction on the received audio data. This process uses an audio processing library (e.g., Audacity or FFmpeg). It takes the decoded audio data as input and applies an algorithm to remove background noise to obtain clear audio output. It also performs volume normalization to ensure that all audio data has a consistent volume level.

[1739] Step 3:

[1740] The server inputs the noise-removed and volume-normalized audio data into a speech recognition engine, which converts it into text data. Specifically, it uses the Google Speech-to-Text or IBM Watson Speech to Text API. The speech recognition engine analyzes the input audio data, converts each syllable and word into text format, and outputs it.

[1741] Step 4:

[1742] The server analyzes the generated text data using natural language processing techniques. This process uses tools such as TextBlob and NLTK to extract emotional information and keywords from the text data. The input is the text data, and the output is an emotional label (positive, negative, neutral) and a list of key keywords.

[1743] Step 5:

[1744] The server uses an emotion engine to analyze emotional trends from both voice and text data. During this process, it analyzes factors such as tone, speed, and intensity of the voice to identify the user's emotional state. It uses text data and voice features as input data and obtains real-time emotional trend information as output.

[1745] Step 6:

[1746] The server evaluates user performance based on the sentiment analysis and keyword extraction results using a machine learning model (using TensorFlow or PyTorch). This machine learning model learns from past data and identifies patterns of high and low performers. It uses the sentiment analysis results and keyword list as input data and generates a performance evaluation score as output.

[1747] Step 7:

[1748] The server generates feedback for the user based on the performance evaluation results, including specific improvements and advice. This feedback also includes the emotional trend obtained by the emotional engine. The server uses the performance evaluation score and the emotional trend information as inputs and generates a feedback message as output.

[1749] Step 8:

[1750] The server sends the generated feedback to the user's device. This data is then encrypted again and sent securely to the user's device. The device receives the feedback data and visually displays it in the user interface. The user can review this feedback and implement improvements in their next business call or sales conversation.

[1751] (Application example 2)

[1752] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1753] In modern factories, cooperation between workers and machines is extremely important. However, when workers are under high stress or tension, communication efficiency declines, which can have a negative impact on productivity and safety. Current technology lacks a mechanism for recognizing workers' emotional state in real time and providing immediate, specific feedback. Therefore, there is a need for a system that can manage workers' stress and tension, maintain smooth communication, and improve work efficiency.

[1754] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1755] In this invention, the server includes means for receiving voice data, means for noise reduction and volume normalization of the voice data, means for converting the voice data into text data, means for performing sentiment analysis on the text data, means for evaluating performance based on the sentiment analysis and keyword extraction results, means for generating specific feedback based on the evaluation results and providing the feedback to the user, and means for supporting communication between factory workers and machines in real time and providing feedback based on the worker's emotional state. This makes it possible to grasp the worker's emotional state in real time and provide prompt and appropriate feedback.

[1756] "Audio data" refers to data that represents audio in digital form.

[1757] "Noise reduction" refers to the process of removing unnecessary noise from audio data.

[1758] "Volume normalization" refers to the process of adjusting the volume of audio data to a constant level.

[1759] "Text data" refers to data obtained by converting voice data into characters.

[1760] "Sentiment analysis" is a technology that identifies emotional states from text and audio data.

[1761] "Keyword extraction" is the process of extracting important words and phrases from text data.

[1762] "Performance evaluation" refers to evaluating a user's behavior and abilities based on the results of sentiment analysis and keyword extraction.

[1763] "Feedback" refers to suggestions for improvement and advice provided to users based on the evaluation results.

[1764] "Workers" refers to personnel who perform work within a factory.

[1765] "Machinery" refers to various devices and robots used within the factory.

[1766] "Real-time communication support" means instantly analyzing and supporting the dialogue between workers and machines.

[1767] "Emotional state" refers to the psychological state of a worker, such as stress or tension.

[1768] "Operation guidance" refers to instructing workers on appropriate work methods and countermeasures based on their emotional state.

[1769] The system of this invention supports real-time communication between factory workers and machines, improving work efficiency and safety. The system has three main components: a server, a terminal, and a user.

[1770] Server Functionality Description

[1771] The server is responsible for processing the received audio data. The specific processing flow is as follows:

[1772] 1. Receive audio data:

[1773] The server receives the voice data sent from the device. The received data is securely encrypted.

[1774] 2. Preprocessing the audio data:

[1775] The server performs noise reduction and volume normalization on the received audio data using pydub.

[1776] 3. Transcription of audio data:

[1777] The server uses a speech recognition model (such as Huggingface's Wav2Vec2) to convert the audio data into text.

[1778] 4. Sentiment analysis and keyword extraction:

[1779] The server then uses natural language processing technology and a sentiment analysis engine to analyze sentiment and extract keywords from the converted text data. Sentiment analysis is performed using the Huggingface pipeline.

[1780] 5. Performance evaluation and feedback generation:

[1781] The server evaluates the worker's performance based on the results of emotion analysis and keyword extraction, and generates specific feedback and operational guidance according to the worker's emotional state.

[1782] Device function description

[1783] A terminal is a device that users use in a factory. Its specific operations are as follows:

[1784] 1. Audio recording and transmission:

[1785] The device records the worker's voice in real time and sends the recorded data to a server, where it is temporarily stored and encrypted.

[1786] 2. Receiving and Viewing Feedback:

[1787] The device receives the feedback sent from the server and displays it to the user in an easy-to-understand manner, including operational guidance based on the user's emotional state.

[1788] User behavior description

[1789] Users are primarily factory workers who use the system to improve their own work. Specific examples of actions include:

[1790] 1. Start recording:

[1791] The worker starts recording the voice using the terminal at the same time as starting the work.

[1792] 2. Use feedback:

[1793] After receiving the feedback, the worker follows the instructions and implements the improvements in the next work process. For example, if tension is detected during work, the worker follows specific advice such as "take a deep breath and relax."

[1794] Examples and prompts

[1795] As a concrete example, consider the following scenario: a factory worker issues a warning about a malfunctioning machine:

[1796] Worker voice recording: "There is an abnormal noise in the machine's bearings, please check for oil leaks on the right side."

[1797] Sentiment analysis results: Detected that the worker was nervous.

[1798] Feedback: "We are in a tense situation, so please refer to the emergency response checklist for when an emergency occurs."

[1799] An example of a prompt for a generative AI model: "There is an abnormal noise in the machine's bearings, so please check the right side for oil leaks."

[1800] In this way, the system of the present invention can recognize the emotional state of workers in real time and provide appropriate feedback, thereby improving the quality of communication within the factory and significantly improving work efficiency and safety.

[1801] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1802] Step 1:

[1803] Audio recording and transmission

[1804] Description: The device records the worker's voice in real time and sends the recorded data to the server.

[1805] Input: Worker's voice input.

[1806] Output: Encrypted audio data.

[1807] How it works: The device records the worker's voice in real time and temporarily stores the data. The stored data is encrypted and sent to a server using a secure protocol (e.g., HTTPS).

[1808] Step 2:

[1809] Audio data preprocessing

[1810] Description: The server performs noise reduction and volume normalization on the received audio data.

[1811] Input: Encrypted audio data received from the device.

[1812] Output: Denoised and volume-normalised audio data.

[1813] Specific operation: The server decrypts the received audio data based on the encryption method, and uses the pydub library to perform noise removal and volume normalization.

[1814] Step 3:

[1815] Transcription of audio data

[1816] Description: The server uses a speech recognition model to convert the audio data into text data.

[1817] Input: Denoised and volume-normalised audio data.

[1818] Output: Text data.

[1819] Specific operation: The server inputs the noise-removed and volume-normalized audio data into a speech recognition model (e.g., Huggingface's Wav2Vec2) and converts the audio into corresponding text data.

[1820] Step 4:

[1821] Sentiment analysis and keyword extraction

[1822] Description: The server performs sentiment analysis and keyword extraction on the converted text data.

[1823] Input: Transcribed text data.

[1824] Output: Emotional state and keyword extraction results.

[1825] Specific operation: The server inputs text data into the Huggingface pipeline and performs sentiment analysis and keyword extraction. The obtained emotional state and keywords are used to evaluate performance in the next step.

[1826] Step 5:

[1827] Performance evaluation and feedback generation

[1828] Description: The server evaluates the worker's performance based on the results of sentiment analysis and keyword extraction, and generates specific feedback and operation guidance.

[1829] Input: Emotional state and keyword extraction results.

[1830] Output: Specific feedback and operation guidance.

[1831] Specific Action: The server integrates the results of sentiment analysis and keyword extraction to evaluate the worker's overall performance, including stress and tension. The generated feedback includes specific improvements and operational guidance.

[1832] Step 6:

[1833] Receiving and viewing feedback

[1834] Description: The device receives feedback sent from the server and displays it visually to the user.

[1835] Input: Feedback received from the server.

[1836] Output: Feedback information that is displayed to the user.

[1837] Specific operation: The device asynchronously receives feedback sent from the server and displays it in an easy-to-understand visualization format for the user, allowing the user to check work improvements and emergency response measures in real time.

[1838] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1839] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1840] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1841] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1842] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1843] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1844] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1845] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1846] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1847] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1848] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1849] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1850] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1851] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1852] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1853] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1854] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1855] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1856] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1857] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1858] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1859] The following is further disclosed regarding the above embodiment.

[1860] (Claim 1)

[1861] means for receiving audio data;

[1862] means for denoising and volume normalising the audio data;

[1863] means for converting the voice data into text data;

[1864] means for performing sentiment analysis on the text data;

[1865] a means for evaluating performance based on the sentiment analysis and keyword extraction results;

[1866] means for generating specific feedback based on the evaluation results and providing the feedback to the user;

[1867] A system including:

[1868] (Claim 2)

[1869] means for transmitting the recorded voice data to a server;

[1870] means for receiving feedback from the server and displaying it to a user;

[1871] 10. The system of claim 1.

[1872] (Claim 3)

[1873] means for recording user voice data in real time and temporarily saving the recorded data at regular intervals;

[1874] means for encrypting and securely transmitting the recorded data to a server;

[1875] 10. The system of claim 1.

[1876] "Example 1"

[1877] (Claim 1)

[1878] means for receiving audio data;

[1879] means for denoising and volume normalising the audio data;

[1880] means for converting the voice data into text data;

[1881] means for performing sentiment analysis and keyword extraction on the text data;

[1882] a means for evaluating performance based on the sentiment analysis and keyword extraction results;

[1883] means for generating specific feedback based on the evaluation results and providing the feedback to the user;

[1884] means for visualizing said feedback through a user interface;

[1885] A system including:

[1886] (Claim 2)

[1887] means for recording and transmitting audio data to a server;

[1888] means for receiving feedback from the server and visualizing and displaying the feedback to a user;

[1889] 10. The system of claim 1.

[1890] (Claim 3)

[1891] means for recording user voice data in real time and temporarily saving the recorded data at regular intervals;

[1892] means for encrypting and securely transmitting the recorded data to a server;

[1893] 10. The system of claim 1.

[1894] "Application Example 1"

[1895] (Claim 1)

[1896] means for receiving audio data;

[1897] means for denoising and volume normalising the audio data;

[1898] means for converting the voice data into text data;

[1899] means for performing sentiment analysis on the text data;

[1900] a means for evaluating performance based on the sentiment analysis and keyword extraction results;

[1901] means for generating specific feedback based on the evaluation results and providing the feedback to the user;

[1902] A method for analyzing and optimizing voice data in real time so that robots can give appropriate work instructions and status reports.

[1903] A system including:

[1904] (Claim 2)

[1905] means for transmitting the recorded voice data to a server;

[1906] means for receiving feedback from the server and displaying it to a user;

[1907] 10. The system of claim 1.

[1908] (Claim 3)

[1909] means for recording user voice data in real time and temporarily saving the recorded data at regular intervals;

[1910] means for encrypting and securely transmitting the recorded data to a server;

[1911] 10. The system of claim 1.

[1912] "Example 2: Combining Emotion Engines"

[1913] (Claim 1)

[1914] means for receiving voice data from a user terminal;

[1915] means for denoising and volume normalising the audio data;

[1916] means for converting the voice data into text data;

[1917] means for performing sentiment analysis and keyword extraction on the text data using natural language processing technology;

[1918] means for analyzing sentiment trends from both the voice data and the text data;

[1919] a means for evaluating performance based on the sentiment analysis and keyword extraction results;

[1920] means for generating specific feedback based on the evaluation results and providing the feedback to the user;

[1921] means for transmitting the feedback to a user terminal;

[1922] A system including:

[1923] (Claim 2)

[1924] means for transmitting the recorded voice data to a server;

[1925] means for receiving feedback from the server and displaying it to a user;

[1926] 10. The system of claim 1.

[1927] (Claim 3)

[1928] means for recording user voice data in real time and temporarily saving the recorded data at regular intervals;

[1929] means for encrypting the recorded data and transmitting it securely to a server;

[1930] 10. The system of claim 1.

[1931] "Application example 2 when combining emotion engines"

[1932] (Claim 1)

[1933] means for receiving audio data;

[1934] means for denoising and volume normalising the audio data;

[1935] means for converting the voice data into text data;

[1936] means for performing sentiment analysis on the text data;

[1937] a means for evaluating performance based on the sentiment analysis and keyword extraction results;

[1938] means for generating specific feedback based on the evaluation results and providing the feedback to the user;

[1939] A means to support real-time communication between factory workers and machines, providing feedback based on emotional state; and

[1940] A system including:

[1941] (Claim 2)

[1942] means for transmitting the recorded voice data to a server;

[1943] means for receiving feedback from the server and displaying it to a user;

[1944] 10. The system of claim 1.

[1945] (Claim 3)

[1946] means for recording user voice data in real time and temporarily saving the recorded data at regular intervals;

[1947] means for encrypting and securely transmitting the recorded data to a server;

[1948] A means for detecting stress or tension of a factory worker based on emotion analysis and providing operational guidance.

[1949] 10. The system of claim 1. [Explanation of symbols]

[1950] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving audio data; means for denoising and volume normalising the audio data; means for converting the voice data into text data; means for performing sentiment analysis on the text data; a means for evaluating performance based on the sentiment analysis and keyword extraction results; means for generating specific feedback based on the evaluation results and providing the feedback to the user; A system including:

2. means for transmitting the recorded voice data to a server; means for receiving feedback from the server and displaying it to a user; The system of claim 1 .

3. means for recording user voice data in real time and temporarily saving the recorded data at regular intervals; means for encrypting and securely transmitting the recorded data to a server; The system of claim 1 .

Images