System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting audio; means for pre-processing and performing noise reduction and loudness normalisation; means for converting the collected audio into textual data using a generative AI model; means for summarizing the generated textual data and extracting key information; means for storing the summarized data in cloud storage; and means for displaying real-time feedback.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional speech recognition technology has struggled to achieve high accuracy due to a lack of data on Japanese speakers and regional differences. Furthermore, the development of speech analysis AI requires a large amount of data, and the lack of data specialized for the Japanese language has hindered further technological improvement. This has led to problems such as inefficient minutes creation and sharing of know-how during corporate meetings and sales calls. It is necessary to solve these issues and achieve high-quality speech recognition and transcription. [Means for solving the problem]

[0005] The present invention solves these problems with a system that includes a means for collecting voice data, a means for performing preprocessing to remove noise and normalize the volume, a means for converting the collected voice data into text data using a generative AI model, a means for summarizing the generated text data and extracting important information, a means for storing the summarized data in cloud storage, and a means for displaying real-time feedback. Furthermore, by including a means for dividing the preprocessed voice data into chunks, high-precision analysis can be performed. By including a means for linking with voice analysis devices used in face-to-face meetings and sales situations, efficient voice analysis can be achieved in a wide range of usage scenarios.

[0006] "Audio data" refers to data in which audio is recorded in digital format.

[0007] "Preprocessing" is the process of performing initial processing on audio data, such as noise removal and volume normalization.

[0008] "Noise reduction" is a process of removing unnecessary background sounds and noise from audio data.

[0009] "Volume normalization" is the process of adjusting the volume of audio data to a consistent level.

[0010] A "generative AI model" is an artificial intelligence model trained using machine learning or deep learning, and is capable of automatically performing specific tasks.

[0011] "Text data" is character information expressed as language by analyzing voice data.

[0012] A "summary" is a short summary that extracts important information or key points from long text or audio data.

[0013] "Cloud storage" is a storage system for saving and managing data via the Internet.

[0014] "Real-time feedback" is a function that instantly displays results and progress to the user while processing or analysis is taking place.

[0015] "Chunking" is the process of dividing audio data into small, manageable blocks.

[0016] An "audio analysis device" is a hardware device for collecting and analyzing audio data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a speech analysis system that collects speech data and analyzes and summarizes it with high accuracy. Each component cooperates to achieve efficient and accurate speech analysis. Below, we will explain specific examples of processing at each stage of this system.

[0039] Audio data collection

[0040] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through a microphone.

[0041] For example, if a user says, "Let's have our next meeting on Tuesday next week," this speech is immediately collected by the terminal.

[0042] Audio data preprocessing

[0043] Once the server receives the collected audio data, it performs noise reduction and volume normalization, making the audio data suitable for analysis.

[0044] For example, if background noise or echo is present, it will be removed, leaving the audio data clear.

[0045] Speech recognition and transcription

[0046] The server uses a generative AI model to convert the preprocessed audio data into text data, which then generates highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language.

[0047] For example, a user's statement "Let's have the next meeting on Tuesday next week" is accurately output as text data as "Let's have the next meeting on Tuesday next week."

[0048] Summarizing text data

[0049] The server analyzes the generated text data, extracts and summarizes important information, and efficiently organizes the main points and important comments from the meeting.

[0050] For example, a long text data of an entire meeting can be summarized as "The next meeting will be held next Tuesday."

[0051] Data storage and sharing

[0052] The server stores the summarized data in cloud storage and makes it accessible to users through a web portal, allowing them to easily view and share the data they need.

[0053] For example, after a meeting, a user can log in to a portal site, download the minutes, and share them with team members.

[0054] Real-time feedback

[0055] The device displays real-time transcription and analysis results, allowing users to keep track of progress during the meeting and receive feedback as needed.

[0056] For example, important statements made during a meeting are converted into text in real time and displayed on the screen.

[0057] Voice analysis device integration

[0058] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations, making voice analysis possible even in offline environments, and allowing for efficient recording.

[0059] For example, a user can use a dedicated voice analysis device at a sales site, and the contents of business negotiations conducted there will be converted into text in real time and saved.

[0060] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, helping to improve the efficiency of communication both inside and outside the company.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer.

[0064] Step 2:

[0065] The device periodically sends the collected voice data to a server, allowing the data to be analyzed in real time.

[0066] Step 3:

[0067] The server passes the received audio data to a pre-processing module for noise reduction and volume normalization. Noise reduction removes background sounds and background noise, while volume normalization maintains consistent audio quality.

[0068] Step 4:

[0069] The server inputs the preprocessed speech data into the generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[0070] Step 5:

[0071] The server analyzes the generated text data and extracts important keywords and phrases, allowing for an efficient and concise summary of the meeting content.

[0072] Step 6:

[0073] The server stores the summarized text data in cloud storage, and the summarized data containing important information is safely stored in the storage.

[0074] Step 7:

[0075] After the meeting, users can access the web portal and download or share the minutes and summary data generated from the cloud storage, making it easier to share information within the company.

[0076] Step 8:

[0077] The device displays the progress of transcription and summarization in real time during the meeting, allowing users to check what is being said in real time and make corrections or additions on the spot as needed.

[0078] Step 9:

[0079] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations, making it possible to efficiently analyze and store voice data on-site.

[0080] The above steps create a system that consistently collects, analyzes, stores, and shares voice data.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] Conventional voice collection and analysis systems have limitations in the accuracy of real-time transcription and analysis, and suffer from insufficient noise reduction and volume normalization. They also face problems with the efficiency of analyzing and summarizing collected voice data. Furthermore, device integration is insufficient when used in face-to-face meetings or sales situations, making voice analysis in offline environments difficult.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes means for collecting audio data, means for performing preprocessing, noise reduction, and volume normalization, means for converting the collected audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in remote storage, means for displaying real-time feedback, and means for analyzing and processing the audio data. This enables highly accurate audio analysis and summarization in real time, allowing for efficient and effective data collection, processing, and sharing.

[0086] "Voice data" means data that is a digital representation of a user's voice.

[0087] "Preprocessing" refers to a series of processes for preparing collected audio data to make it easier to analyze, and includes noise removal and volume normalization.

[0088] "Noise reduction" is a process that removes unnecessary background sounds and noise from audio data.

[0089] "Volume normalization" is a process for making the volume level of audio data uniform.

[0090] A "generative AI model" refers to an algorithm or statistical model that uses artificial intelligence to perform a specific task and is used to convert voice data into text.

[0091] "Text data" is data obtained by converting voice data into a character string.

[0092] "Summarization" refers to concisely summarizing information and includes the process of extracting important information.

[0093] "Remote storage" is a data storage area that can be accessed via a network, such as a cloud environment.

[0094] "Real-time feedback" refers to the function of returning processing results to the user immediately.

[0095] A "voice analysis device" is a device or system dedicated to analyzing voice data.

[0096] The present invention is a voice analysis system in which each component cooperates to achieve efficient and accurate voice analysis. A specific method for implementing this system will be described in detail below.

[0097] Audio data collection

[0098] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time using the device's built-in microphone. The device is often a PC or smartphone. For example, if a user says, "Let's schedule our next meeting for next Tuesday," this audio is captured by the device's microphone.

[0099] Audio data preprocessing

[0100] When audio data is sent from the device to the server, the server uses Python libraries (such as Librosa or Pydub) to remove noise and normalize the volume. This process prepares the audio data for analysis. For example, if background noise or echo is present, it is removed, resulting in clear audio data.

[0101] Speech recognition and transcription

[0102] The server that receives the preprocessed voice data converts it into text data using a generative AI model such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text. This conversion accurately converts the speech into text. For example, a user uttering "Let's have our next meeting on Tuesday next week" is accurately output as text data as "Let's have our next meeting on Tuesday next week."

[0103] Summarizing text data

[0104] The server then analyzes the converted text data using a natural language processing (NLP) library (e.g., Spacy or NLTK) to extract and summarize important information. This allows for efficient extraction of key points from large amounts of text data. For example, a long meeting's minutes can be summarized as "The next meeting will be held next Tuesday."

[0105] Data storage and sharing

[0106] The server stores the summarized data in a remote storage (e.g., AWS S3, Google Cloud Storage), and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the generated minutes, and share them with team members.

[0107] Real-time feedback

[0108] The device displays the transcription and analysis results in real time, allowing users to understand the progress of the meeting and receive immediate feedback as needed. For example, anything said during a meeting is immediately transcribed and displayed on the device screen.

[0109] Linking voice analysis devices

[0110] The device works in conjunction with a voice analysis device used in face-to-face meetings and sales situations. This allows for voice analysis even in offline environments, making record-keeping more efficient. For example, a user can use a dedicated voice analysis device at a sales site to convert the contents of a sales negotiation into text in real time and save it.

[0111] Examples of prompt statements

[0112] Below are some specific examples of prompt sentences to be input to the generative AI model and the results of their processing.

[0113] Prompt statement:

[0114] Please transcribe what is said and then summarize it. Here is the audio: "Let's have our next meeting next Tuesday."

[0115] Processing result:

[0116] Transcription result: Let's have our next meeting next Tuesday.

[0117] Summary result: The next meeting will be next Tuesday.

[0118] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, making it possible to significantly improve the efficiency of communication both inside and outside a company.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1:

[0121] Audio data collection

[0122] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time using the device's built-in microphone. The input is the user's voice obtained through the microphone, and the output is the captured voice data. For example, a user might say, "Let's have our next meeting on Tuesday next week," and this is captured by the device's microphone.

[0123] Step 2:

[0124] Sending audio data to the server

[0125] The terminal sends the captured voice data to the server. The input is the captured voice data obtained in step 1, and the output is the voice data sent to the server. Specifically, the voice data is compressed and converted into packets, and then sent to the server.

[0126] Step 3:

[0127] Audio data preprocessing

[0128] The server uses Python libraries (e.g., Librosa or Pydub) to remove noise and normalize the volume of the audio data it receives. The input is the raw audio data received by the server, and the output is clear audio data that has been preprocessed. Specifically, the Librosa library is used to filter background noise, and the Pydub library is used to equalize the volume level.

[0129] Step 4:

[0130] Transcription of audio data

[0131] The server uses a generative AI model, such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text, to convert the preprocessed audio data into text. The input is clear preprocessed audio data, and the output is text data that details the audio content. Specific operations include sending audio data to the API and receiving a text response.

[0132] Step 5:

[0133] Summarizing text data

[0134] The server analyzes the converted text data using a natural language processing (NLP) library (such as Spacy or NLTK) to extract and summarize important information. The input is the full text data converted from speech, and the output is a summary text that summarizes the important points. Specifically, the Spacy library is used to analyze the content of the text data and extract important phrases and keywords.

[0135] Step 6:

[0136] Data storage and sharing

[0137] The server stores the summarized data in remote storage (e.g., AWS S3 or Google Cloud Storage) and makes it accessible to users through a web portal. The input is the summarized text data, and the output is the data stored in cloud storage. Specifically, a URL and access key for accessing the stored data are generated and linked to the user's account.

[0138] Step 7:

[0139] Providing real-time feedback

[0140] The device displays the transcription and analysis results in real time. The input is the transcription and summary data sent sequentially from the server, and the output is feedback information displayed on the user's screen. Specifically, the user's remarks are transcribed in real time, and the device screen displays, "Let's schedule the next meeting for next Tuesday."

[0141] Step 8:

[0142] Linking voice analysis devices

[0143] The device connects to a voice analysis device used in face-to-face meetings and sales situations. The input is voice data obtained from a dedicated voice analysis device, and the output is text data as the analysis results. Specific operations include analyzing voice data collected offline in real time and saving it on the device.

[0144] (Application example 1)

[0145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] Conventional speech analysis systems have limitations in accuracy and efficiency at each step of speech data collection, preprocessing, transcription, and summarization, making it difficult to provide real-time feedback or effectively utilize analysis results, especially in dynamic environments such as stores. This creates problems such as poor customer service quality and limited employee training effectiveness.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0148] In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in cloud storage, means for displaying real-time feedback, means for analyzing the voice data sent from the voice recognition device, and means for store employees to receive the analysis results in real time. This enables high-precision voice analysis and immediate feedback even in a dynamic store environment, thereby improving the quality of customer service and maximizing the effectiveness of employee training.

[0149] "Audio data" refers to data that is collected and stored in digital form.

[0150] "Preprocessing" refers to the process of removing noise from collected audio data and normalizing the volume to make it suitable for analysis.

[0151] A "generative AI model" is an artificial intelligence model used to convert voice data into text data.

[0152] "Text data" refers to data obtained by converting voice data into text format.

[0153] "Summarization" is the process of extracting important information from the generated text data and summarizing it in a compact format.

[0154] "Cloud storage" is a remote server for storing and managing data over the Internet.

[0155] "Real-time feedback" is a function that instantly conveys the results of voice analysis to users on the spot.

[0156] A "voice recognition device" is a hardware device for collecting and analyzing speech.

[0157] "Analysis" is the process of understanding the content and extracting meaning from collected data.

[0158] "Store employees" are personnel who deal with customers and perform other duties in a store.

[0159] The present invention is a system that collects and analyzes voice data in real time and provides immediate feedback to improve the quality of customer service in physical stores. The following describes an embodiment of the present invention.

[0160] Program Overview

[0161] The server, voice recognition device, and store employee terminals (such as smart glasses) work together.

[0162] Hardware and Software Configuration

[0163] The server is configured using the following software and hardware:

[0164] Voice data collection method: Smart glasses with built-in microphones and portable voice recorders are used as voice recognition devices.

[0165] Preprocessing method: To preprocess the audio data, software for noise reduction and volume normalization (e.g., Audacity) is used.

[0166] Generative AI models: Use generative AI models (e.g., OpenAI Whisper) to convert voice data into text data with high accuracy.

[0167] Summarization: To summarize the generated text data, we use a pipeline using NLP techniques (e.g., Hugging Face Transformers).

[0168] Cloud storage means: Use cloud storage services (e.g., AWS S3) to store summary data.

[0169] Real-time feedback: Develop a custom application to display analysis results in real time on employees' smart glasses.

[0170] System operation explanation

[0171] Audio data collection

[0172] The server receives and collects the voice data sent from the voice recognition device. For example, when a store employee talks to a customer, the microphone built into the smart glasses captures the voice data in real time.

[0173] Audio data preprocessing

[0174] The server removes noise from the received audio data and normalizes the volume, making the data suitable for analysis.

[0175] Speech recognition and transcription

[0176] The server uses the generative AI model to convert the pre-processed speech data into text data. For example, if a customer asks, "Tell me about next week's campaign," the speech is instantly converted into text.

[0177] Summarizing text data

[0178] The server analyzes the text data, extracts important information, and summarizes it. This allows for efficient organization of key points from long conversations. For example, a summary might be generated that reads, "We need to explain the campaign."

[0179] Data storage and sharing

[0180] The server stores the summary data in cloud storage and makes it accessible to store managers as needed.

[0181] Real-time feedback

[0182] The analysis results are displayed in real time on store employees' smart glasses, enabling them to immediately respond appropriately to customer questions.

[0183] Usage examples and prompt statements

[0184] For example, if a customer asks a store employee, "Tell me about next week's campaign," the smart glasses collect this question as voice data and send it to the server. The server preprocesses the voice data and converts it into text data using a generative AI model. The text data is summarized and the message "Please explain the campaign" is displayed on the smart glasses in real time.

[0185] Example prompt sentence:

[0186] "You will create a program that will analyze and summarize customer conversations in real time and display them on smart glasses. The hardware used will be smart glasses, and a generative AI model will be used for speech recognition. The analysis results will be stored in cloud storage."

[0187] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0188] Step 1:

[0189] The user puts on the smart glasses and starts talking to the customer. The microphone built into the smart glasses captures voice data in real time and sends it to the terminal. The input is the conversation between the customer and the user, and the output is the captured raw voice data.

[0190] Step 2:

[0191] The device sends the captured audio data to the server. The input is the raw audio data sent from the device, and the output is the audio data received by the server.

[0192] Step 3:

[0193] The server preprocesses the audio data it receives, removing noise and normalizing the volume to prepare the audio data for analysis. The input is raw audio data, and the output is clear, preprocessed audio data.

[0194] Step 4:

[0195] The server inputs the preprocessed audio data into a generative AI model and converts the audio data into text data. A generative AI model (e.g., OpenAI Whisper) is used to perform highly accurate transcription. The input is the preprocessed audio data, and the output is the generated text data.

[0196] Step 5:

[0197] The server analyzes the generated text data and summarizes it, extracting important information and summarizing the main points in a compact format. The input is text data, and the output is summarized information.

[0198] Step 6:

[0199] The server saves the summarized information in cloud storage. Specifically, it uploads the data to AWS S3, for example. The input is the summarized information, and the output is the data saved in cloud storage.

[0200] Step 7:

[0201] The analysis results are displayed in real time on the user's smart glasses. As the user continues to interact with the customer, summarized information is displayed on the smart glasses' display. The input is the summarized information, and the output is the feedback displayed on the smart glasses' display.

[0202] Step 8:

[0203] The user responds appropriately to customers based on the feedback from the smart glasses. The input is the information displayed on the smart glasses, and the output is to improve the quality of customer service.

[0204] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0205] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[0206] Audio data collection

[0207] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's have our next meeting on Tuesday next week," this voice is immediately collected by the device.

[0208] Audio data preprocessing

[0209] Once the server receives the collected audio data, it performs noise reduction and volume normalization to prepare the audio data for analysis. For example, background noise and echoes are removed to ensure the audio data is clear.

[0210] Speech recognition and transcription

[0211] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language. For example, a user's statement, "Let's have our next meeting on Tuesday of next week," is accurately output as text data, "Let's have our next meeting on Tuesday of next week."

[0212] Summarizing text data

[0213] The server analyzes the generated text data and extracts and summarizes important information. This allows for efficient organization of key points and important remarks from the meeting. For example, a long text data of the entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[0214] Emotion recognition by emotion engine

[0215] The server uses an emotion engine to recognize the user's emotions from the voice data. For example, it can determine emotional states such as "joy," "anger," or "sadness" from the tone and tempo of the user's speech. The recognized emotions are tagged in the text data and saved as emotional information.

[0216] Data storage and sharing

[0217] The server stores the summarized data and emotion information in cloud storage, and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[0218] Real-time feedback

[0219] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0220] Voice analysis device integration

[0221] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[0222] As described above, the voice analysis system of the present invention can consistently perform processes from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, thereby achieving comprehensive analysis that also includes the user's emotions.

[0223] The processing flow will be explained below.

[0224] Step 1:

[0225] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's schedule our next meeting for next Tuesday," the audio is captured by the device.

[0226] Step 2:

[0227] The terminal periodically transmits the buffered voice data to the server, which stores the received voice data in a database and awaits the next processing step.

[0228] Step 3:

[0229] The server performs noise reduction and volume normalization on the audio data through a pre-processing module, which removes background noise and normalizes the volume to a certain level. For example, audio data with a lot of echo or noise can be denoised to produce clear audio data.

[0230] Step 4:

[0231] The server divides the preprocessed speech data into chunks and inputs them into a generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[0232] Step 5:

[0233] The server analyzes the generated text data, extracts and summarizes important information, for example, summarizing a long meeting as "The next meeting will be held on Tuesday next week."

[0234] Step 6:

[0235] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the voice tone, speed, pitch, etc. to identify emotions such as "joy," "anger," or "sadness," and tags them in the text data. For example, if the user speaks with a happy expression, the tag "joy" is added.

[0236] Step 7:

[0237] The server stores the summarized text data and emotion information in cloud storage, where the data is securely stored and can be accessed by the user later.

[0238] Step 8:

[0239] After the meeting, users can access a web portal to download or share the generated minutes and summary data from the cloud storage. For example, they can share emotion-tagged minutes with team members.

[0240] Step 9:

[0241] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0242] Step 10:

[0243] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations. This will enable efficient analysis and storage of voice data in the field. For example, a dedicated voice analysis device will be used in a sales situation, and the content of the sales negotiations and emotional information at the time will be converted into text and stored in real time.

[0244] The above steps create a system that can consistently collect, analyze, convert, and share voice data.

[0245] Example 2

[0246] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0247] Conventional speech analysis systems have issues with noise reduction, volume normalization, data conversion, and summarization in the collection and analysis of speech data, and are particularly lacking in real-time feedback and emotion recognition. Furthermore, collecting and processing speech data in offline environments is difficult, resulting in insufficient user convenience.

[0248] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0249] In this invention, the server includes means for collecting audio data, means for preprocessing the collected audio data to perform noise reduction and volume normalization, means for converting the preprocessed audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the audio data and tagging the generated text data with the recognized emotions, means for storing the summarized data and emotional information in cloud storage, means for allowing users to access the stored data via a web portal, and means for displaying real-time feedback. This enables a consistent process from audio data collection to analysis, data conversion, emotion recognition, storage, sharing, and real-time feedback.

[0250] "Means for collecting voice data" refers to a function that captures the voice spoken by a user during a meeting or face-to-face conversation using a voice input device such as a microphone on the device, and temporarily stores that voice.

[0251] "Noise removal" is a process that removes unnecessary sounds such as background noise and echo from collected voice data, making it suitable for voice analysis.

[0252] "Volume normalization" is a process that adjusts the volume of collected audio data to a constant level in order to improve the accuracy of analysis and text conversion.

[0253] A "generative AI model" is an artificial intelligence algorithm or program that has the ability to convert voice data into text data, and is specifically designed to achieve highly accurate voice recognition.

[0254] "Means for converting into text data" refers to a function that converts preprocessed audio data into text format data using a generative AI model.

[0255] The "means for summarizing text data" is a function that extracts important information from the generated text data and summarizes the main points in a concise manner.

[0256] The "means for recognizing emotions" is a function that analyzes the user's emotions from voice data, identifies those emotions, and generates appropriate emotion tags.

[0257] The "means for tagging the generated text data" is a function that adds the recognized emotion to the text data as a tag, making the text data easier to search and reference later.

[0258] "Means for saving to cloud storage" is a function that saves summarized text data and its emotion tags on an external cloud server, making it easy to store and share data.

[0259] "Means for allowing users to access via a web portal" means a web-based interface that allows users to access data stored in cloud storage via the Internet.

[0260] The "means for displaying real-time feedback" is a function that displays information such as analyzed text data, its summary, and emotion tags on the user's device in real time.

[0261] The "means for dividing into chunks" is a function for dividing preprocessed audio data into small data blocks at regular intervals for efficient analysis and processing.

[0262] "Means for linking with voice analysis devices" refers to a mechanism that connects to dedicated voice analysis equipment used in face-to-face meetings and sales situations, and provides the function of collecting and analyzing data.

[0263] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[0264] Audio data collection

[0265] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through the microphone and temporarily stores it in a buffer. Specifically, a built-in microphone or an external microphone device can be used. For example, if a user says, "Let's have our next meeting on Tuesday next week," this audio is immediately collected by the device.

[0266] Audio data preprocessing

[0267] Once the server receives the collected audio data, it performs noise reduction and volume normalization using advanced noise filtering algorithms and volume adjustment software, for example, removing background noise and echo to ensure clear audio data.

[0268] Speech recognition and transcription

[0269] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences in the Japanese language and the diversity of pronunciation. For example, a user's statement, "Let's have our next meeting on Tuesday next week," is accurately output as text data as, "Let's have our next meeting on Tuesday next week."

[0270] Summarizing text data

[0271] The server analyzes the generated text data, extracts and summarizes important information, typically using natural language processing algorithms. For example, a long text data entry for an entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[0272] Emotion recognition by emotion engine

[0273] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes parameters such as the tone and tempo of the voice to determine emotions such as "happiness," "anger," and "sadness." For example, "happiness" is recognized from the tone and tempo of the voice when the user speaks, and an emotion tag such as "happiness" is added to the text data.

[0274] Data storage and sharing

[0275] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The cloud storage service can use any popular online storage solution. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[0276] Real-time feedback

[0277] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0278] Voice analysis device integration

[0279] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[0280] As described above, the voice analysis system of the present invention can consistently perform everything from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, and realizes comprehensive analysis including the user's emotions. This allows users to obtain more accurate and useful information in meetings and face-to-face communications.

[0281] Specific examples

[0282] Example prompt: "Let's schedule our next meeting for next Tuesday." Explain how this system will collect and analyze data and provide feedback to the user.

[0283] Generated description: If a user says, "Let's schedule our next meeting for next Tuesday," the device collects the audio data, and the server performs noise reduction and volume normalization. The generative AI model then converts the audio into text, and the text data is summarized as "Let's schedule our next meeting for next Tuesday." The emotion engine then recognizes this as "joy," and when the user logs in to the portal, they can view the emotion-tagged minutes.

[0284] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0285] Program processing steps

[0286] Step 1: Collecting audio data

[0287] A user participates in a web conferencing system or a face-to-face meeting. The device captures voice data in real time through the microphone and temporarily stores it in a buffer. The input is the user's voice, and the output is the voice data temporarily stored in the buffer. For example, when a user says, "Let's have our next meeting on Tuesday next week," the device's microphone captures the voice, and the data is temporarily stored on the device.

[0288] Step 2: Preprocessing the audio data

[0289] The server receives audio data from the device. It performs noise removal and volume normalization on the received audio data. The input is the audio data stored in the buffer, and the output is the preprocessed audio data. Specifically, background noise and echo are removed from the audio data, and the volume is adjusted to a constant level.

[0290] Step 3: Speech recognition and transcription

[0291] The server inputs preprocessed voice data into the generative AI model. The generative AI model converts the voice data into text data. The input is preprocessed voice data, and the output is text data. Specifically, the voice saying "Let's have our next meeting on Tuesday next week" is converted into text by the generative AI model as "Let's have our next meeting on Tuesday next week."

[0292] Step 4: Summarizing the text data

[0293] The server analyzes the generated text data and extracts and summarizes important information. The input is the generated text data, and the output is the summarized text data. Specifically, the text data is analyzed by a natural language processing algorithm and summarized as "The next meeting will be held on Tuesday next week."

[0294] Step 5: Emotion Recognition with the Emotion Engine

[0295] The server uses an emotion engine to recognize the user's emotion from the voice data. The recognized emotion is then tagged to the text data. The input is voice data and text data, and the output is text data with an emotion tag. Specifically, "joy" is recognized from the voice tone and tempo, and a "joy" tag is added to the text data.

[0296] Step 6: Store and share data

[0297] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The input is emotion-tagged text data, and the output is data stored in cloud storage. Specifically, users can log in to the portal site after the meeting and download the emotion-tagged minutes.

[0298] Step 7: Real-time feedback

[0299] The device displays the progress of transcription and summarization in real time during the meeting. Emotional information recognized by the emotion engine is also displayed in real time. The input is text data and emotional information, and the output is real-time display information that can be checked by the user. Specifically, during the meeting, the message "Let's have our next meeting on Tuesday next week" is displayed, and the "joy" tag is also displayed at the same time.

[0300] Step 8: Linking your voice analysis device

[0301] The terminal works in conjunction with a voice analysis device that is also used in face-to-face meetings and sales situations. This makes it possible to collect and analyze voice data even in offline environments. The input is offline voice data, and the output is text-translated and emotion-tagged data. Specifically, the user uses a dedicated voice analysis device at the sales site, and the content of the sales negotiations and emotional information held there are converted into text in real time and saved.

[0302] (Application example 2)

[0303] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0304] Conventional voice analysis systems were capable of transcribing general voice data and removing noise, but were unable to recognize the speaker's emotional state in real time and provide feedback based on that. This made it difficult for staff to understand the customer's emotions in customer service or sales situations, making it difficult to respond quickly and appropriately. The objective of the present invention is to solve these problems and provide a system that can understand the content and emotions of customers in real time and support appropriate responses.

[0305] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the voice data using an emotion engine, means for saving the summarized data and emotion information in cloud storage, means for displaying real-time feedback, and means for displaying feedback in real time using smart glasses to instantly grasp the customer's comments and emotions. This allows staff to grasp the content of customer comments and emotions in real time and respond promptly and appropriately.

[0306] "Audio data" refers to data that represents an audio signal in digital form.

[0307] "Means for collecting" refers to a method or device for capturing audio data using a microphone or other input device.

[0308] "Preprocessing" refers to the process of removing noise from audio data and normalizing the volume for high-quality analysis.

[0309] "Noise reduction" is a process for removing unnecessary background sounds from audio data to obtain a clear audio signal.

[0310] "Volume normalization" is a process for making the volume level of audio data constant.

[0311] A "generative AI model" is an algorithm or system that uses machine learning technology to analyze voice data with high accuracy.

[0312] "Text data" refers to audio data that has been converted into text and is displayed in an easy-to-read format.

[0313] A "summarizing means" is a method or device for extracting important information from the generated text data and summarizing it concisely.

[0314] An "emotion engine" is a system that analyzes and recognizes people's emotions from the tone of their voice and the content of their statements.

[0315] "Emotion information" is data about the speaker's emotional state obtained from the voice data analyzed by the emotion engine.

[0316] "Cloud storage" is a collection of remote servers that store and access data over the internet.

[0317] "Real-time feedback" is a function that instantly processes voice data and analysis results and provides information to the user immediately.

[0318] "Smart glasses" are glasses-type devices with display functions that are used by the wearer to obtain visual information.

[0319] The "means for displaying feedback" refers to a method or device for providing analysis results or summary information to a user through an output device such as a display.

[0320] "Customer utterances" are messages or questions expressed by customers through speech.

[0321] "Customer service" refers to a series of activities to provide information and services in response to customer needs.

[0322] This invention is a system that supports customer service in brick-and-mortar stores, collecting, preprocessing, transcribing, summarizing, and recognizing emotions from voice data in real time, allowing staff to instantly understand the content and emotions of customers and respond appropriately.

[0323] Hardware and software used

[0324] Hardware

[0325] Smart glasses: Collects voice data in real time and displays the analysis results on a screen.

[0326] Server: Responsible for preprocessing, transcription, summarization, and emotion recognition of audio data.

[0327] software

[0328] Speech Recognition Library (speech_recognition): Converts voice data into text data.

[0329] Noise Reduction Library (noisereduce): Performs noise reduction and volume normalization.

[0330] Generative AI model (openai API): A generative AI model for summarization.

[0331] Emotion Recognizer: Recognizes emotions from audio data.

[0332] Data processing and calculation

[0333] 1. Collection of audio data

[0334] A microphone built into the smart glasses collects customer speech and records it as audio data, which is then sent to a server in real time.

[0335] 2. Pretreatment

[0336] The server removes noise and normalizes the volume of the received audio data to prepare it for analysis.

[0337] 3. Transcription

[0338] The preprocessed speech data is converted into text data using a speech recognition library. For example, a statement such as "I would like to know if new products will be available next month" is accurately converted into text.

[0339] 4. Summarizing Text Data

[0340] The server uses a generative AI model to extract and summarize important information from the generated text data, for example, "Confirm the arrival schedule of new products."

[0341] 5. Emotion recognition

[0342] An emotion recognition engine is used on text data to recognize customer emotions. For example, the emotion "interest" is recognized.

[0343] 6. Real-time feedback

[0344] The smart glasses display summary and sentiment information in real time, allowing staff to instantly understand customer needs and emotions.

[0345] Specific examples

[0346] For example, if a customer says, "I'd like to know if new products will be arriving next month," this speech is collected by the smart glasses' microphone. The audio data is sent to a server where noise is removed and volume normalized. It is then transcribed using a speech recognition library, generating text data such as "I'd like to know if new products will be arriving next month." This text data is then summarized using a generative AI model as "Check the new product arrival schedule," and the emotion recognition engine recognizes the emotion "interest." Finally, this information is displayed in real time on the smart glasses' display, allowing staff to quickly respond to the customer.

[0347] Prompt Sentence Examples

[0348] An example prompt for a generative AI model is:

[0349] Summarize the full text below:

[0350] I would like to know if any new products will be in stock next month.

[0351] summary:

[0352] The above is an embodiment of the present invention, which allows staff to grasp the content of customer comments and emotions in real time, enabling them to respond quickly and appropriately.

[0353] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0354] Step 1:

[0355] Audio data collection

[0356] Input: User (customer) utterance

[0357] Specific operation: The user speaks through the microphone built into the smart glasses. The smart glasses capture the user's speech as audio data in real time and temporarily store the data in the smart glasses.

[0358] Output: Recorded audio data

[0359] Step 2:

[0360] Audio data preprocessing

[0361] Input: Recorded audio data

[0362] How it works: Smart glasses collect and send audio data to a server, which then performs noise reduction and volume normalization on the received audio data to remove background noise and echoes and maintain a consistent volume level.

[0363] Output: Pre-processed clear audio data

[0364] Step 3:

[0365] Transcription

[0366] Input: Preprocessed, clear audio data

[0367] Specific operation: The server converts the preprocessed audio data into text data using a speech recognition library (speech_recognition), which analyzes the audio signal and generates a corresponding string of characters.

[0368] Output: Generated text data (e.g., "I would like to know if new products will be available next month.")

[0369] Step 4:

[0370] Summarizing text data

[0371] Input: Generated text data

[0372] Specific operation: The server uses a generative AI model (openai API) to extract and summarize important information from text data. The generative AI model receives a specific prompt (e.g., "Please summarize the following sentence: I would like to know if new products will be available next month.") and generates a summarized text.

[0373] Output: Summarized text data (e.g., "Confirm new product arrival schedule")

[0374] Step 5:

[0375] emotion recognition

[0376] Input: Generated text data

[0377] Specific operation: The server uses an emotion recognition engine (EmotionRecognizer) to extract emotional information from text data. The emotion recognition engine analyzes the emotional elements contained in the text and recognizes the emotional state (e.g., interest).

[0378] Output: Recognized emotion information (e.g., "interest")

[0379] Step 6:

[0380] View real-time feedback

[0381] Input: Summarized text data and recognized emotion information

[0382] How it works: The server sends summarized text data and emotional information to the smart glasses, which then display it in real time on the display for the user to check immediately.

[0383] Output: Real-time summary information and emotion information (e.g., "Check new product arrival schedule" and "Interested" are displayed on the screen)

[0384] Step 7:

[0385] Data storage and sharing

[0386] Input: Summarized text data and sentiment information

[0387] How it works: The server stores the summarized text data and emotion information in cloud storage, which can be accessed later and downloaded by users via a web portal to share with team members as needed.

[0388] Output: Text data and emotion information stored in cloud storage

[0389] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0390] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0391] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0392] [Second embodiment]

[0393] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0394] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0395] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0396] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0397] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0398] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0399] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0400] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0401] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0402] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0403] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0404] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0405] This invention is a speech analysis system that collects speech data and analyzes and summarizes it with high accuracy. Each component cooperates to achieve efficient and accurate speech analysis. Below, we will explain specific examples of processing at each stage of this system.

[0406] Audio data collection

[0407] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through a microphone.

[0408] For example, if a user says, "Let's have our next meeting on Tuesday next week," this speech is immediately collected by the terminal.

[0409] Audio data preprocessing

[0410] Once the server receives the collected audio data, it performs noise reduction and volume normalization, making the audio data suitable for analysis.

[0411] For example, if background noise or echo is present, it will be removed, leaving the audio data clear.

[0412] Speech recognition and transcription

[0413] The server uses a generative AI model to convert the preprocessed audio data into text data, which then generates highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language.

[0414] For example, a user's statement "Let's have the next meeting on Tuesday next week" is accurately output as text data as "Let's have the next meeting on Tuesday next week."

[0415] Summarizing text data

[0416] The server analyzes the generated text data, extracts and summarizes important information, and efficiently organizes the main points and important comments from the meeting.

[0417] For example, a long text data of an entire meeting can be summarized as "The next meeting will be held next Tuesday."

[0418] Data storage and sharing

[0419] The server stores the summarized data in cloud storage and makes it accessible to users through a web portal, allowing them to easily view and share the data they need.

[0420] For example, after a meeting, a user can log in to a portal site, download the minutes, and share them with team members.

[0421] Real-time feedback

[0422] The device displays real-time transcription and analysis results, allowing users to keep track of progress during the meeting and receive feedback as needed.

[0423] For example, important statements made during a meeting are converted into text in real time and displayed on the screen.

[0424] Voice analysis device integration

[0425] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations, making voice analysis possible even in offline environments, and allowing for efficient recording.

[0426] For example, a user can use a dedicated voice analysis device at a sales site, and the contents of business negotiations conducted there will be converted into text in real time and saved.

[0427] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, helping to improve the efficiency of communication both inside and outside the company.

[0428] The processing flow will be explained below.

[0429] Step 1:

[0430] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer.

[0431] Step 2:

[0432] The device periodically sends the collected voice data to a server, allowing the data to be analyzed in real time.

[0433] Step 3:

[0434] The server passes the received audio data to a pre-processing module for noise reduction and volume normalization. Noise reduction removes background sounds and background noise, while volume normalization maintains consistent audio quality.

[0435] Step 4:

[0436] The server inputs the preprocessed speech data into the generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[0437] Step 5:

[0438] The server analyzes the generated text data and extracts important keywords and phrases, allowing for an efficient and concise summary of the meeting content.

[0439] Step 6:

[0440] The server stores the summarized text data in cloud storage, and the summarized data containing important information is safely stored in the storage.

[0441] Step 7:

[0442] After the meeting, users can access the web portal and download or share the minutes and summary data generated from the cloud storage, making it easier to share information within the company.

[0443] Step 8:

[0444] The device displays the progress of transcription and summarization in real time during the meeting, allowing users to check what is being said in real time and make corrections or additions on the spot as needed.

[0445] Step 9:

[0446] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations, making it possible to efficiently analyze and store voice data on-site.

[0447] The above steps create a system that consistently collects, analyzes, stores, and shares voice data.

[0448] Example 1

[0449] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0450] Conventional voice collection and analysis systems have limitations in the accuracy of real-time transcription and analysis, and suffer from insufficient noise reduction and volume normalization. They also face problems with the efficiency of analyzing and summarizing collected voice data. Furthermore, device integration is insufficient when used in face-to-face meetings or sales situations, making voice analysis in offline environments difficult.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0452] In this invention, the server includes means for collecting audio data, means for performing preprocessing, noise reduction, and volume normalization, means for converting the collected audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in remote storage, means for displaying real-time feedback, and means for analyzing and processing the audio data. This enables highly accurate audio analysis and summarization in real time, allowing for efficient and effective data collection, processing, and sharing.

[0453] "Voice data" means data that is a digital representation of a user's voice.

[0454] "Preprocessing" refers to a series of processes for preparing collected audio data to make it easier to analyze, and includes noise removal and volume normalization.

[0455] "Noise reduction" is a process that removes unnecessary background sounds and noise from audio data.

[0456] "Volume normalization" is a process for making the volume level of audio data uniform.

[0457] A "generative AI model" refers to an algorithm or statistical model that uses artificial intelligence to perform a specific task and is used to convert voice data into text.

[0458] "Text data" is data obtained by converting voice data into a character string.

[0459] "Summarization" refers to concisely summarizing information and includes the process of extracting important information.

[0460] "Remote storage" is a data storage area that can be accessed via a network, such as a cloud environment.

[0461] "Real-time feedback" refers to the function of returning processing results to the user immediately.

[0462] A "voice analysis device" is a device or system dedicated to analyzing voice data.

[0463] The present invention is a voice analysis system in which each component cooperates to achieve efficient and accurate voice analysis. A specific method for implementing this system will be described in detail below.

[0464] Audio data collection

[0465] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time using the device's built-in microphone. The device is often a PC or smartphone. For example, if a user says, "Let's schedule our next meeting for next Tuesday," this audio is captured by the device's microphone.

[0466] Audio data preprocessing

[0467] When audio data is sent from the device to the server, the server uses Python libraries (such as Librosa or Pydub) to remove noise and normalize the volume. This process prepares the audio data for analysis. For example, if background noise or echo is present, it is removed, resulting in clear audio data.

[0468] Speech recognition and transcription

[0469] The server that receives the preprocessed voice data converts it into text data using a generative AI model such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text. This conversion accurately converts the speech into text. For example, a user uttering "Let's have our next meeting on Tuesday next week" is accurately output as text data as "Let's have our next meeting on Tuesday next week."

[0470] Summarizing text data

[0471] The server then analyzes the converted text data using a natural language processing (NLP) library (e.g., Spacy or NLTK) to extract and summarize important information. This allows for efficient extraction of key points from large amounts of text data. For example, a long meeting's minutes can be summarized as "The next meeting will be held next Tuesday."

[0472] Data storage and sharing

[0473] The server stores the summarized data in a remote storage (e.g., AWS S3, Google Cloud Storage), and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the generated minutes, and share them with team members.

[0474] Real-time feedback

[0475] The device displays the transcription and analysis results in real time, allowing users to understand the progress of the meeting and receive immediate feedback as needed. For example, anything said during a meeting is immediately transcribed and displayed on the device screen.

[0476] Linking voice analysis devices

[0477] The device works in conjunction with a voice analysis device used in face-to-face meetings and sales situations. This allows for voice analysis even in offline environments, making record-keeping more efficient. For example, a user can use a dedicated voice analysis device at a sales site to convert the contents of a sales negotiation into text in real time and save it.

[0478] Examples of prompt statements

[0479] Below are some specific examples of prompt sentences to be input to the generative AI model and the results of their processing.

[0480] Prompt statement:

[0481] Please transcribe what is said and then summarize it. Here is the audio: "Let's have our next meeting next Tuesday."

[0482] Processing result:

[0483] Transcription result: Let's have our next meeting next Tuesday.

[0484] Summary result: The next meeting will be next Tuesday.

[0485] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, making it possible to significantly improve the efficiency of communication both inside and outside a company.

[0486] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0487] Step 1:

[0488] Audio data collection

[0489] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time using the device's built-in microphone. The input is the user's voice obtained through the microphone, and the output is the captured voice data. For example, a user might say, "Let's have our next meeting on Tuesday next week," and this is captured by the device's microphone.

[0490] Step 2:

[0491] Sending audio data to the server

[0492] The terminal sends the captured voice data to the server. The input is the captured voice data obtained in step 1, and the output is the voice data sent to the server. Specifically, the voice data is compressed and converted into packets, and then sent to the server.

[0493] Step 3:

[0494] Audio data preprocessing

[0495] The server uses Python libraries (e.g., Librosa or Pydub) to remove noise and normalize the volume of the audio data it receives. The input is the raw audio data received by the server, and the output is clear audio data that has been preprocessed. Specifically, the Librosa library is used to filter background noise, and the Pydub library is used to equalize the volume level.

[0496] Step 4:

[0497] Transcription of audio data

[0498] The server uses a generative AI model, such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text, to convert the preprocessed audio data into text. The input is clear preprocessed audio data, and the output is text data that details the audio content. Specific operations include sending audio data to the API and receiving a text response.

[0499] Step 5:

[0500] Summarizing text data

[0501] The server analyzes the converted text data using a natural language processing (NLP) library (such as Spacy or NLTK) to extract and summarize important information. The input is the full text data converted from speech, and the output is a summary text that summarizes the important points. Specifically, the Spacy library is used to analyze the content of the text data and extract important phrases and keywords.

[0502] Step 6:

[0503] Data storage and sharing

[0504] The server stores the summarized data in remote storage (e.g., AWS S3 or Google Cloud Storage) and makes it accessible to users through a web portal. The input is the summarized text data, and the output is the data stored in cloud storage. Specifically, a URL and access key for accessing the stored data are generated and linked to the user's account.

[0505] Step 7:

[0506] Providing real-time feedback

[0507] The device displays the transcription and analysis results in real time. The input is the transcription and summary data sent sequentially from the server, and the output is feedback information displayed on the user's screen. Specifically, the user's remarks are transcribed in real time, and the device screen displays, "Let's schedule the next meeting for next Tuesday."

[0508] Step 8:

[0509] Linking voice analysis devices

[0510] The device connects to a voice analysis device used in face-to-face meetings and sales situations. The input is voice data obtained from a dedicated voice analysis device, and the output is text data as the analysis results. Specific operations include analyzing voice data collected offline in real time and saving it on the device.

[0511] (Application example 1)

[0512] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0513] Conventional speech analysis systems have limitations in accuracy and efficiency at each step of speech data collection, preprocessing, transcription, and summarization, making it difficult to provide real-time feedback or effectively utilize analysis results, especially in dynamic environments such as stores. This creates problems such as poor customer service quality and limited employee training effectiveness.

[0514] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0515] In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in cloud storage, means for displaying real-time feedback, means for analyzing the voice data sent from the voice recognition device, and means for store employees to receive the analysis results in real time. This enables high-precision voice analysis and immediate feedback even in a dynamic store environment, thereby improving the quality of customer service and maximizing the effectiveness of employee training.

[0516] "Audio data" refers to data that is collected and stored in digital form.

[0517] "Preprocessing" refers to the process of removing noise from collected audio data and normalizing the volume to make it suitable for analysis.

[0518] A "generative AI model" is an artificial intelligence model used to convert voice data into text data.

[0519] "Text data" refers to data obtained by converting voice data into text format.

[0520] "Summarization" is the process of extracting important information from the generated text data and summarizing it in a compact format.

[0521] "Cloud storage" is a remote server for storing and managing data over the Internet.

[0522] "Real-time feedback" is a function that instantly conveys the results of voice analysis to users on the spot.

[0523] A "voice recognition device" is a hardware device for collecting and analyzing speech.

[0524] "Analysis" is the process of understanding the content and extracting meaning from collected data.

[0525] "Store employees" are personnel who deal with customers and perform other duties in a store.

[0526] The present invention is a system that collects and analyzes voice data in real time and provides immediate feedback to improve the quality of customer service in physical stores. The following describes an embodiment of the present invention.

[0527] Program Overview

[0528] The server, voice recognition device, and store employee terminals (such as smart glasses) work together.

[0529] Hardware and Software Configuration

[0530] The server is configured using the following software and hardware:

[0531] Voice data collection method: Smart glasses with built-in microphones and portable voice recorders are used as voice recognition devices.

[0532] Preprocessing method: To preprocess the audio data, software for noise reduction and volume normalization (e.g., Audacity) is used.

[0533] Generative AI models: Use generative AI models (e.g., OpenAI Whisper) to convert voice data into text data with high accuracy.

[0534] Summarization: To summarize the generated text data, we use a pipeline using NLP techniques (e.g., Hugging Face Transformers).

[0535] Cloud storage means: Use cloud storage services (e.g., AWS S3) to store summary data.

[0536] Real-time feedback: Develop a custom application to display analysis results in real time on employees' smart glasses.

[0537] System operation explanation

[0538] Audio data collection

[0539] The server receives and collects the voice data sent from the voice recognition device. For example, when a store employee talks to a customer, the microphone built into the smart glasses captures the voice data in real time.

[0540] Audio data preprocessing

[0541] The server removes noise from the received audio data and normalizes the volume, making the data suitable for analysis.

[0542] Speech recognition and transcription

[0543] The server uses the generative AI model to convert the pre-processed speech data into text data. For example, if a customer asks, "Tell me about next week's campaign," the speech is instantly converted into text.

[0544] Summarizing text data

[0545] The server analyzes the text data, extracts important information, and summarizes it. This allows for efficient organization of key points from long conversations. For example, a summary might be generated that reads, "We need to explain the campaign."

[0546] Data storage and sharing

[0547] The server stores the summary data in cloud storage and makes it accessible to store managers as needed.

[0548] Real-time feedback

[0549] The analysis results are displayed in real time on store employees' smart glasses, enabling them to immediately respond appropriately to customer questions.

[0550] Usage examples and prompt statements

[0551] For example, if a customer asks a store employee, "Tell me about next week's campaign," the smart glasses collect this question as voice data and send it to the server. The server preprocesses the voice data and converts it into text data using a generative AI model. The text data is summarized and the message "Please explain the campaign" is displayed on the smart glasses in real time.

[0552] Example prompt sentence:

[0553] "You will create a program that will analyze and summarize customer conversations in real time and display them on smart glasses. The hardware used will be smart glasses, and a generative AI model will be used for speech recognition. The analysis results will be stored in cloud storage."

[0554] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0555] Step 1:

[0556] The user puts on the smart glasses and starts talking to the customer. The microphone built into the smart glasses captures voice data in real time and sends it to the terminal. The input is the conversation between the customer and the user, and the output is the captured raw voice data.

[0557] Step 2:

[0558] The device sends the captured audio data to the server. The input is the raw audio data sent from the device, and the output is the audio data received by the server.

[0559] Step 3:

[0560] The server preprocesses the audio data it receives, removing noise and normalizing the volume to prepare the audio data for analysis. The input is raw audio data, and the output is clear, preprocessed audio data.

[0561] Step 4:

[0562] The server inputs the preprocessed audio data into a generative AI model and converts the audio data into text data. A generative AI model (e.g., OpenAI Whisper) is used to perform highly accurate transcription. The input is the preprocessed audio data, and the output is the generated text data.

[0563] Step 5:

[0564] The server analyzes the generated text data and summarizes it, extracting important information and summarizing the main points in a compact format. The input is text data, and the output is summarized information.

[0565] Step 6:

[0566] The server saves the summarized information in cloud storage. Specifically, it uploads the data to AWS S3, for example. The input is the summarized information, and the output is the data saved in cloud storage.

[0567] Step 7:

[0568] The analysis results are displayed in real time on the user's smart glasses. As the user continues to interact with the customer, summarized information is displayed on the smart glasses' display. The input is the summarized information, and the output is the feedback displayed on the smart glasses' display.

[0569] Step 8:

[0570] The user responds appropriately to customers based on the feedback from the smart glasses. The input is the information displayed on the smart glasses, and the output is to improve the quality of customer service.

[0571] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0572] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[0573] Audio data collection

[0574] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's have our next meeting on Tuesday next week," this voice is immediately collected by the device.

[0575] Audio data preprocessing

[0576] Once the server receives the collected audio data, it performs noise reduction and volume normalization to prepare the audio data for analysis. For example, background noise and echoes are removed to ensure the audio data is clear.

[0577] Speech recognition and transcription

[0578] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language. For example, a user's statement, "Let's have our next meeting on Tuesday of next week," is accurately output as text data, "Let's have our next meeting on Tuesday of next week."

[0579] Summarizing text data

[0580] The server analyzes the generated text data and extracts and summarizes important information. This allows for efficient organization of key points and important remarks from the meeting. For example, a long text data of the entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[0581] Emotion recognition by emotion engine

[0582] The server uses an emotion engine to recognize the user's emotions from the voice data. For example, it can determine emotional states such as "joy," "anger," or "sadness" from the tone and tempo of the user's speech. The recognized emotions are tagged in the text data and saved as emotional information.

[0583] Data storage and sharing

[0584] The server stores the summarized data and emotion information in cloud storage, and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[0585] Real-time feedback

[0586] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0587] Voice analysis device integration

[0588] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[0589] As described above, the voice analysis system of the present invention can consistently perform processes from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, thereby achieving comprehensive analysis that also includes the user's emotions.

[0590] The processing flow will be explained below.

[0591] Step 1:

[0592] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's schedule our next meeting for next Tuesday," the audio is captured by the device.

[0593] Step 2:

[0594] The terminal periodically transmits the buffered voice data to the server, which stores the received voice data in a database and awaits the next processing step.

[0595] Step 3:

[0596] The server performs noise reduction and volume normalization on the audio data through a pre-processing module, which removes background noise and normalizes the volume to a certain level. For example, audio data with a lot of echo or noise can be denoised to produce clear audio data.

[0597] Step 4:

[0598] The server divides the preprocessed speech data into chunks and inputs them into a generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[0599] Step 5:

[0600] The server analyzes the generated text data, extracts and summarizes important information, for example, summarizing a long meeting as "The next meeting will be held on Tuesday next week."

[0601] Step 6:

[0602] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the voice tone, speed, pitch, etc. to identify emotions such as "joy," "anger," or "sadness," and tags them in the text data. For example, if the user speaks with a happy expression, the tag "joy" is added.

[0603] Step 7:

[0604] The server stores the summarized text data and emotion information in cloud storage, where the data is securely stored and can be accessed by the user later.

[0605] Step 8:

[0606] After the meeting, users can access a web portal to download or share the generated minutes and summary data from the cloud storage. For example, they can share emotion-tagged minutes with team members.

[0607] Step 9:

[0608] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0609] Step 10:

[0610] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations. This will enable efficient analysis and storage of voice data in the field. For example, a dedicated voice analysis device will be used in a sales situation, and the content of the sales negotiations and emotional information at the time will be converted into text and stored in real time.

[0611] The above steps create a system that can consistently collect, analyze, convert, and share voice data.

[0612] Example 2

[0613] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] Conventional speech analysis systems have issues with noise reduction, volume normalization, data conversion, and summarization in the collection and analysis of speech data, and are particularly lacking in real-time feedback and emotion recognition. Furthermore, collecting and processing speech data in offline environments is difficult, resulting in insufficient user convenience.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0616] In this invention, the server includes means for collecting audio data, means for preprocessing the collected audio data to perform noise reduction and volume normalization, means for converting the preprocessed audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the audio data and tagging the generated text data with the recognized emotions, means for storing the summarized data and emotional information in cloud storage, means for allowing users to access the stored data via a web portal, and means for displaying real-time feedback. This enables a consistent process from audio data collection to analysis, data conversion, emotion recognition, storage, sharing, and real-time feedback.

[0617] "Means for collecting voice data" refers to a function that captures the voice spoken by a user during a meeting or face-to-face conversation using a voice input device such as a microphone on the device, and temporarily stores that voice.

[0618] "Noise removal" is a process that removes unnecessary sounds such as background noise and echo from collected voice data, making it suitable for voice analysis.

[0619] "Volume normalization" is a process that adjusts the volume of collected audio data to a constant level in order to improve the accuracy of analysis and text conversion.

[0620] A "generative AI model" is an artificial intelligence algorithm or program that has the ability to convert voice data into text data, and is specifically designed to achieve highly accurate voice recognition.

[0621] "Means for converting into text data" refers to a function that converts preprocessed audio data into text format data using a generative AI model.

[0622] The "means for summarizing text data" is a function that extracts important information from the generated text data and summarizes the main points in a concise manner.

[0623] The "means for recognizing emotions" is a function that analyzes the user's emotions from voice data, identifies those emotions, and generates appropriate emotion tags.

[0624] The "means for tagging the generated text data" is a function that adds the recognized emotion to the text data as a tag, making the text data easier to search and reference later.

[0625] "Means for saving to cloud storage" is a function that saves summarized text data and its emotion tags on an external cloud server, making it easy to store and share data.

[0626] "Means for allowing users to access via a web portal" means a web-based interface that allows users to access data stored in cloud storage via the Internet.

[0627] The "means for displaying real-time feedback" is a function that displays information such as analyzed text data, its summary, and emotion tags on the user's device in real time.

[0628] The "means for dividing into chunks" is a function for dividing preprocessed audio data into small data blocks at regular intervals for efficient analysis and processing.

[0629] "Means for linking with voice analysis devices" refers to a mechanism that connects to dedicated voice analysis equipment used in face-to-face meetings and sales situations, and provides the function of collecting and analyzing data.

[0630] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[0631] Audio data collection

[0632] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through the microphone and temporarily stores it in a buffer. Specifically, a built-in microphone or an external microphone device can be used. For example, if a user says, "Let's have our next meeting on Tuesday next week," this audio is immediately collected by the device.

[0633] Audio data preprocessing

[0634] Once the server receives the collected audio data, it performs noise reduction and volume normalization using advanced noise filtering algorithms and volume adjustment software, for example, removing background noise and echo to ensure clear audio data.

[0635] Speech recognition and transcription

[0636] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences in the Japanese language and the diversity of pronunciation. For example, a user's statement, "Let's have our next meeting on Tuesday next week," is accurately output as text data as, "Let's have our next meeting on Tuesday next week."

[0637] Summarizing text data

[0638] The server analyzes the generated text data, extracts and summarizes important information, typically using natural language processing algorithms. For example, a long text data entry for an entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[0639] Emotion recognition by emotion engine

[0640] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes parameters such as the tone and tempo of the voice to determine emotions such as "happiness," "anger," and "sadness." For example, "happiness" is recognized from the tone and tempo of the voice when the user speaks, and an emotion tag such as "happiness" is added to the text data.

[0641] Data storage and sharing

[0642] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The cloud storage service can use any popular online storage solution. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[0643] Real-time feedback

[0644] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0645] Voice analysis device integration

[0646] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[0647] As described above, the voice analysis system of the present invention can consistently perform everything from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, and realizes comprehensive analysis including the user's emotions. This allows users to obtain more accurate and useful information in meetings and face-to-face communications.

[0648] Specific examples

[0649] Example prompt: "Let's schedule our next meeting for next Tuesday." Explain how this system will collect and analyze data and provide feedback to the user.

[0650] Generated description: If a user says, "Let's schedule our next meeting for next Tuesday," the device collects the audio data, and the server performs noise reduction and volume normalization. The generative AI model then converts the audio into text, and the text data is summarized as "Let's schedule our next meeting for next Tuesday." The emotion engine then recognizes this as "joy," and when the user logs in to the portal, they can view the emotion-tagged minutes.

[0651] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0652] Program processing steps

[0653] Step 1: Collecting audio data

[0654] A user participates in a web conferencing system or a face-to-face meeting. The device captures voice data in real time through the microphone and temporarily stores it in a buffer. The input is the user's voice, and the output is the voice data temporarily stored in the buffer. For example, when a user says, "Let's have our next meeting on Tuesday next week," the device's microphone captures the voice, and the data is temporarily stored on the device.

[0655] Step 2: Preprocessing the audio data

[0656] The server receives audio data from the device. It performs noise removal and volume normalization on the received audio data. The input is the audio data stored in the buffer, and the output is the preprocessed audio data. Specifically, background noise and echo are removed from the audio data, and the volume is adjusted to a constant level.

[0657] Step 3: Speech recognition and transcription

[0658] The server inputs preprocessed voice data into the generative AI model. The generative AI model converts the voice data into text data. The input is preprocessed voice data, and the output is text data. Specifically, the voice saying "Let's have our next meeting on Tuesday next week" is converted into text by the generative AI model as "Let's have our next meeting on Tuesday next week."

[0659] Step 4: Summarizing the text data

[0660] The server analyzes the generated text data and extracts and summarizes important information. The input is the generated text data, and the output is the summarized text data. Specifically, the text data is analyzed by a natural language processing algorithm and summarized as "The next meeting will be held on Tuesday next week."

[0661] Step 5: Emotion Recognition with the Emotion Engine

[0662] The server uses an emotion engine to recognize the user's emotion from the voice data. The recognized emotion is then tagged to the text data. The input is voice data and text data, and the output is text data with an emotion tag. Specifically, "joy" is recognized from the voice tone and tempo, and a "joy" tag is added to the text data.

[0663] Step 6: Store and share data

[0664] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The input is emotion-tagged text data, and the output is data stored in cloud storage. Specifically, users can log in to the portal site after the meeting and download the emotion-tagged minutes.

[0665] Step 7: Real-time feedback

[0666] The device displays the progress of transcription and summarization in real time during the meeting. Emotional information recognized by the emotion engine is also displayed in real time. The input is text data and emotional information, and the output is real-time display information that can be checked by the user. Specifically, during the meeting, the message "Let's have our next meeting on Tuesday next week" is displayed, and the "joy" tag is also displayed at the same time.

[0667] Step 8: Linking your voice analysis device

[0668] The terminal works in conjunction with a voice analysis device that is also used in face-to-face meetings and sales situations. This makes it possible to collect and analyze voice data even in offline environments. The input is offline voice data, and the output is text-translated and emotion-tagged data. Specifically, the user uses a dedicated voice analysis device at the sales site, and the content of the sales negotiations and emotional information held there are converted into text in real time and saved.

[0669] (Application example 2)

[0670] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] Conventional voice analysis systems were capable of transcribing general voice data and removing noise, but were unable to recognize the speaker's emotional state in real time and provide feedback based on that. This made it difficult for staff to understand the customer's emotions in customer service or sales situations, making it difficult to respond quickly and appropriately. The objective of the present invention is to solve these problems and provide a system that can understand the content and emotions of customers in real time and support appropriate responses.

[0672] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the voice data using an emotion engine, means for saving the summarized data and emotion information in cloud storage, means for displaying real-time feedback, and means for displaying feedback in real time using smart glasses to instantly grasp the customer's comments and emotions. This allows staff to grasp the content of customer comments and emotions in real time and respond promptly and appropriately.

[0673] "Audio data" refers to data that represents an audio signal in digital form.

[0674] "Means for collecting" refers to a method or device for capturing audio data using a microphone or other input device.

[0675] "Preprocessing" refers to the process of removing noise from audio data and normalizing the volume for high-quality analysis.

[0676] "Noise reduction" is a process for removing unnecessary background sounds from audio data to obtain a clear audio signal.

[0677] "Volume normalization" is a process for making the volume level of audio data constant.

[0678] A "generative AI model" is an algorithm or system that uses machine learning technology to analyze voice data with high accuracy.

[0679] "Text data" refers to audio data that has been converted into text and is displayed in an easy-to-read format.

[0680] A "summarizing means" is a method or device for extracting important information from the generated text data and summarizing it concisely.

[0681] An "emotion engine" is a system that analyzes and recognizes people's emotions from the tone of their voice and the content of their statements.

[0682] "Emotion information" is data about the speaker's emotional state obtained from the voice data analyzed by the emotion engine.

[0683] "Cloud storage" is a collection of remote servers that store and access data over the internet.

[0684] "Real-time feedback" is a function that instantly processes voice data and analysis results and provides information to the user immediately.

[0685] "Smart glasses" are glasses-type devices with display functions that are used by the wearer to obtain visual information.

[0686] The "means for displaying feedback" refers to a method or device for providing analysis results or summary information to a user through an output device such as a display.

[0687] "Customer utterances" are messages or questions expressed by customers through speech.

[0688] "Customer service" refers to a series of activities to provide information and services in response to customer needs.

[0689] This invention is a system that supports customer service in brick-and-mortar stores, collecting, preprocessing, transcribing, summarizing, and recognizing emotions from voice data in real time, allowing staff to instantly understand the content and emotions of customers and respond appropriately.

[0690] Hardware and software used

[0691] Hardware

[0692] Smart glasses: Collects voice data in real time and displays the analysis results on a screen.

[0693] Server: Responsible for preprocessing, transcription, summarization, and emotion recognition of audio data.

[0694] software

[0695] Speech Recognition Library (speech_recognition): Converts voice data into text data.

[0696] Noise Reduction Library (noisereduce): Performs noise reduction and volume normalization.

[0697] Generative AI model (openai API): A generative AI model for summarization.

[0698] Emotion Recognizer: Recognizes emotions from audio data.

[0699] Data processing and calculation

[0700] 1. Collection of audio data

[0701] A microphone built into the smart glasses collects customer speech and records it as audio data, which is then sent to a server in real time.

[0702] 2. Pretreatment

[0703] The server removes noise and normalizes the volume of the received audio data to prepare it for analysis.

[0704] 3. Transcription

[0705] The preprocessed speech data is converted into text data using a speech recognition library. For example, a statement such as "I would like to know if new products will be available next month" is accurately converted into text.

[0706] 4. Summarizing Text Data

[0707] The server uses a generative AI model to extract and summarize important information from the generated text data, for example, "Confirm the arrival schedule of new products."

[0708] 5. Emotion recognition

[0709] An emotion recognition engine is used on text data to recognize customer emotions. For example, the emotion "interest" is recognized.

[0710] 6. Real-time feedback

[0711] The smart glasses display summary and sentiment information in real time, allowing staff to instantly understand customer needs and emotions.

[0712] Specific examples

[0713] For example, if a customer says, "I'd like to know if new products will be arriving next month," this speech is collected by the smart glasses' microphone. The audio data is sent to a server where noise is removed and volume normalized. It is then transcribed using a speech recognition library, generating text data such as "I'd like to know if new products will be arriving next month." This text data is then summarized using a generative AI model as "Check the new product arrival schedule," and the emotion recognition engine recognizes the emotion "interest." Finally, this information is displayed in real time on the smart glasses' display, allowing staff to quickly respond to the customer.

[0714] Prompt Sentence Examples

[0715] An example prompt for a generative AI model is:

[0716] Summarize the full text below:

[0717] I would like to know if any new products will be in stock next month.

[0718] summary:

[0719] The above is an embodiment of the present invention, which allows staff to grasp the content of customer comments and emotions in real time, enabling them to respond quickly and appropriately.

[0720] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0721] Step 1:

[0722] Audio data collection

[0723] Input: User (customer) utterance

[0724] Specific operation: The user speaks through the microphone built into the smart glasses. The smart glasses capture the user's speech as audio data in real time and temporarily store the data in the smart glasses.

[0725] Output: Recorded audio data

[0726] Step 2:

[0727] Audio data preprocessing

[0728] Input: Recorded audio data

[0729] How it works: Smart glasses collect and send audio data to a server, which then performs noise reduction and volume normalization on the received audio data to remove background noise and echoes and maintain a consistent volume level.

[0730] Output: Pre-processed clear audio data

[0731] Step 3:

[0732] Transcription

[0733] Input: Preprocessed, clear audio data

[0734] Specific operation: The server converts the preprocessed audio data into text data using a speech recognition library (speech_recognition), which analyzes the audio signal and generates a corresponding string of characters.

[0735] Output: Generated text data (e.g., "I would like to know if new products will be available next month.")

[0736] Step 4:

[0737] Summarizing text data

[0738] Input: Generated text data

[0739] Specific operation: The server uses a generative AI model (openai API) to extract and summarize important information from text data. The generative AI model receives a specific prompt (e.g., "Please summarize the following sentence: I would like to know if new products will be available next month.") and generates a summarized text.

[0740] Output: Summarized text data (e.g., "Confirm new product arrival schedule")

[0741] Step 5:

[0742] emotion recognition

[0743] Input: Generated text data

[0744] Specific operation: The server uses an emotion recognition engine (EmotionRecognizer) to extract emotional information from text data. The emotion recognition engine analyzes the emotional elements contained in the text and recognizes the emotional state (e.g., interest).

[0745] Output: Recognized emotion information (e.g., "interest")

[0746] Step 6:

[0747] View real-time feedback

[0748] Input: Summarized text data and recognized emotion information

[0749] How it works: The server sends summarized text data and emotional information to the smart glasses, which then display it in real time on the display for the user to check immediately.

[0750] Output: Real-time summary information and emotion information (e.g., "Check new product arrival schedule" and "Interested" are displayed on the screen)

[0751] Step 7:

[0752] Data storage and sharing

[0753] Input: Summarized text data and sentiment information

[0754] How it works: The server stores the summarized text data and emotion information in cloud storage, which can be accessed later and downloaded by users via a web portal to share with team members as needed.

[0755] Output: Text data and emotion information stored in cloud storage

[0756] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0757] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0758] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0759] [Third embodiment]

[0760] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0761] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0762] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0763] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0764] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0765] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0766] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0767] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0768] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0769] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0770] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0771] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0772] This invention is a speech analysis system that collects speech data and analyzes and summarizes it with high accuracy. Each component cooperates to achieve efficient and accurate speech analysis. Below, we will explain specific examples of processing at each stage of this system.

[0773] Audio data collection

[0774] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through a microphone.

[0775] For example, if a user says, "Let's have our next meeting on Tuesday next week," this speech is immediately collected by the terminal.

[0776] Audio data preprocessing

[0777] Once the server receives the collected audio data, it performs noise reduction and volume normalization, making the audio data suitable for analysis.

[0778] For example, if background noise or echo is present, it will be removed, leaving the audio data clear.

[0779] Speech recognition and transcription

[0780] The server uses a generative AI model to convert the preprocessed audio data into text data, which then generates highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language.

[0781] For example, a user's statement "Let's have the next meeting on Tuesday next week" is accurately output as text data as "Let's have the next meeting on Tuesday next week."

[0782] Summarizing text data

[0783] The server analyzes the generated text data, extracts and summarizes important information, and efficiently organizes the main points and important comments from the meeting.

[0784] For example, a long text data of an entire meeting can be summarized as "The next meeting will be held next Tuesday."

[0785] Data storage and sharing

[0786] The server stores the summarized data in cloud storage and makes it accessible to users through a web portal, allowing them to easily view and share the data they need.

[0787] For example, after a meeting, a user can log in to a portal site, download the minutes, and share them with team members.

[0788] Real-time feedback

[0789] The device displays real-time transcription and analysis results, allowing users to keep track of progress during the meeting and receive feedback as needed.

[0790] For example, important statements made during a meeting are converted into text in real time and displayed on the screen.

[0791] Voice analysis device integration

[0792] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations, making voice analysis possible even in offline environments, and allowing for efficient recording.

[0793] For example, a user can use a dedicated voice analysis device at a sales site, and the contents of business negotiations conducted there will be converted into text in real time and saved.

[0794] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, helping to improve the efficiency of communication both inside and outside the company.

[0795] The processing flow will be explained below.

[0796] Step 1:

[0797] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer.

[0798] Step 2:

[0799] The device periodically sends the collected voice data to a server, allowing the data to be analyzed in real time.

[0800] Step 3:

[0801] The server passes the received audio data to a pre-processing module for noise reduction and volume normalization. Noise reduction removes background sounds and background noise, while volume normalization maintains consistent audio quality.

[0802] Step 4:

[0803] The server inputs the preprocessed speech data into the generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[0804] Step 5:

[0805] The server analyzes the generated text data and extracts important keywords and phrases, allowing for an efficient and concise summary of the meeting content.

[0806] Step 6:

[0807] The server stores the summarized text data in cloud storage, and the summarized data containing important information is safely stored in the storage.

[0808] Step 7:

[0809] After the meeting, users can access the web portal and download or share the minutes and summary data generated from the cloud storage, making it easier to share information within the company.

[0810] Step 8:

[0811] The device displays the progress of transcription and summarization in real time during the meeting, allowing users to check what is being said in real time and make corrections or additions on the spot as needed.

[0812] Step 9:

[0813] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations, making it possible to efficiently analyze and store voice data on-site.

[0814] The above steps create a system that consistently collects, analyzes, stores, and shares voice data.

[0815] Example 1

[0816] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0817] Conventional voice collection and analysis systems have limitations in the accuracy of real-time transcription and analysis, and suffer from insufficient noise reduction and volume normalization. They also face problems with the efficiency of analyzing and summarizing collected voice data. Furthermore, device integration is insufficient when used in face-to-face meetings or sales situations, making voice analysis in offline environments difficult.

[0818] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0819] In this invention, the server includes means for collecting audio data, means for performing preprocessing, noise reduction, and volume normalization, means for converting the collected audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in remote storage, means for displaying real-time feedback, and means for analyzing and processing the audio data. This enables highly accurate audio analysis and summarization in real time, allowing for efficient and effective data collection, processing, and sharing.

[0820] "Voice data" means data that is a digital representation of a user's voice.

[0821] "Preprocessing" refers to a series of processes for preparing collected audio data to make it easier to analyze, and includes noise removal and volume normalization.

[0822] "Noise reduction" is a process that removes unnecessary background sounds and noise from audio data.

[0823] "Volume normalization" is a process for making the volume level of audio data uniform.

[0824] A "generative AI model" refers to an algorithm or statistical model that uses artificial intelligence to perform a specific task and is used to convert voice data into text.

[0825] "Text data" is data obtained by converting voice data into a character string.

[0826] "Summarization" refers to concisely summarizing information and includes the process of extracting important information.

[0827] "Remote storage" is a data storage area that can be accessed via a network, such as a cloud environment.

[0828] "Real-time feedback" refers to the function of returning processing results to the user immediately.

[0829] A "voice analysis device" is a device or system dedicated to analyzing voice data.

[0830] The present invention is a voice analysis system in which each component cooperates to achieve efficient and accurate voice analysis. A specific method for implementing this system will be described in detail below.

[0831] Audio data collection

[0832] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time using the device's built-in microphone. The device is often a PC or smartphone. For example, if a user says, "Let's schedule our next meeting for next Tuesday," this audio is captured by the device's microphone.

[0833] Audio data preprocessing

[0834] When audio data is sent from the device to the server, the server uses Python libraries (such as Librosa or Pydub) to remove noise and normalize the volume. This process prepares the audio data for analysis. For example, if background noise or echo is present, it is removed, resulting in clear audio data.

[0835] Speech recognition and transcription

[0836] The server that receives the preprocessed voice data converts it into text data using a generative AI model such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text. This conversion accurately converts the speech into text. For example, a user uttering "Let's have our next meeting on Tuesday next week" is accurately output as text data as "Let's have our next meeting on Tuesday next week."

[0837] Summarizing text data

[0838] The server then analyzes the converted text data using a natural language processing (NLP) library (e.g., Spacy or NLTK) to extract and summarize important information. This allows for efficient extraction of key points from large amounts of text data. For example, a long meeting's minutes can be summarized as "The next meeting will be held next Tuesday."

[0839] Data storage and sharing

[0840] The server stores the summarized data in a remote storage (e.g., AWS S3, Google Cloud Storage), and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the generated minutes, and share them with team members.

[0841] Real-time feedback

[0842] The device displays the transcription and analysis results in real time, allowing users to understand the progress of the meeting and receive immediate feedback as needed. For example, anything said during a meeting is immediately transcribed and displayed on the device screen.

[0843] Linking voice analysis devices

[0844] The device works in conjunction with a voice analysis device used in face-to-face meetings and sales situations. This allows for voice analysis even in offline environments, making record-keeping more efficient. For example, a user can use a dedicated voice analysis device at a sales site to convert the contents of a sales negotiation into text in real time and save it.

[0845] Examples of prompt statements

[0846] Below are some specific examples of prompt sentences to be input to the generative AI model and the results of their processing.

[0847] Prompt statement:

[0848] Please transcribe what is said and then summarize it. Here is the audio: "Let's have our next meeting next Tuesday."

[0849] Processing result:

[0850] Transcription result: Let's have our next meeting next Tuesday.

[0851] Summary result: The next meeting will be next Tuesday.

[0852] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, making it possible to significantly improve the efficiency of communication both inside and outside a company.

[0853] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0854] Step 1:

[0855] Audio data collection

[0856] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time using the device's built-in microphone. The input is the user's voice obtained through the microphone, and the output is the captured voice data. For example, a user might say, "Let's have our next meeting on Tuesday next week," and this is captured by the device's microphone.

[0857] Step 2:

[0858] Sending audio data to the server

[0859] The terminal sends the captured voice data to the server. The input is the captured voice data obtained in step 1, and the output is the voice data sent to the server. Specifically, the voice data is compressed and converted into packets, and then sent to the server.

[0860] Step 3:

[0861] Audio data preprocessing

[0862] The server uses Python libraries (e.g., Librosa or Pydub) to remove noise and normalize the volume of the audio data it receives. The input is the raw audio data received by the server, and the output is clear audio data that has been preprocessed. Specifically, the Librosa library is used to filter background noise, and the Pydub library is used to equalize the volume level.

[0863] Step 4:

[0864] Transcription of audio data

[0865] The server uses a generative AI model, such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text, to convert the preprocessed audio data into text. The input is clear preprocessed audio data, and the output is text data that details the audio content. Specific operations include sending audio data to the API and receiving a text response.

[0866] Step 5:

[0867] Summarizing text data

[0868] The server analyzes the converted text data using a natural language processing (NLP) library (such as Spacy or NLTK) to extract and summarize important information. The input is the full text data converted from speech, and the output is a summary text that summarizes the important points. Specifically, the Spacy library is used to analyze the content of the text data and extract important phrases and keywords.

[0869] Step 6:

[0870] Data storage and sharing

[0871] The server stores the summarized data in remote storage (e.g., AWS S3 or Google Cloud Storage) and makes it accessible to users through a web portal. The input is the summarized text data, and the output is the data stored in cloud storage. Specifically, a URL and access key for accessing the stored data are generated and linked to the user's account.

[0872] Step 7:

[0873] Providing real-time feedback

[0874] The device displays the transcription and analysis results in real time. The input is the transcription and summary data sent sequentially from the server, and the output is feedback information displayed on the user's screen. Specifically, the user's remarks are transcribed in real time, and the device screen displays, "Let's schedule the next meeting for next Tuesday."

[0875] Step 8:

[0876] Linking voice analysis devices

[0877] The device connects to a voice analysis device used in face-to-face meetings and sales situations. The input is voice data obtained from a dedicated voice analysis device, and the output is text data as the analysis results. Specific operations include analyzing voice data collected offline in real time and saving it on the device.

[0878] (Application example 1)

[0879] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0880] Conventional speech analysis systems have limitations in accuracy and efficiency at each step of speech data collection, preprocessing, transcription, and summarization, making it difficult to provide real-time feedback or effectively utilize analysis results, especially in dynamic environments such as stores. This creates problems such as poor customer service quality and limited employee training effectiveness.

[0881] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0882] In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in cloud storage, means for displaying real-time feedback, means for analyzing the voice data sent from the voice recognition device, and means for store employees to receive the analysis results in real time. This enables high-precision voice analysis and immediate feedback even in a dynamic store environment, thereby improving the quality of customer service and maximizing the effectiveness of employee training.

[0883] "Audio data" refers to data that is collected and stored in digital form.

[0884] "Preprocessing" refers to the process of removing noise from collected audio data and normalizing the volume to make it suitable for analysis.

[0885] A "generative AI model" is an artificial intelligence model used to convert voice data into text data.

[0886] "Text data" refers to data obtained by converting voice data into text format.

[0887] "Summarization" is the process of extracting important information from the generated text data and summarizing it in a compact format.

[0888] "Cloud storage" is a remote server for storing and managing data over the Internet.

[0889] "Real-time feedback" is a function that instantly conveys the results of voice analysis to users on the spot.

[0890] A "voice recognition device" is a hardware device for collecting and analyzing speech.

[0891] "Analysis" is the process of understanding the content and extracting meaning from collected data.

[0892] "Store employees" are personnel who deal with customers and perform other duties in a store.

[0893] The present invention is a system that collects and analyzes voice data in real time and provides immediate feedback to improve the quality of customer service in physical stores. The following describes an embodiment of the present invention.

[0894] Program Overview

[0895] The server, voice recognition device, and store employee terminals (such as smart glasses) work together.

[0896] Hardware and Software Configuration

[0897] The server is configured using the following software and hardware:

[0898] Voice data collection method: Smart glasses with built-in microphones and portable voice recorders are used as voice recognition devices.

[0899] Preprocessing method: To preprocess the audio data, software for noise reduction and volume normalization (e.g., Audacity) is used.

[0900] Generative AI models: Use generative AI models (e.g., OpenAI Whisper) to convert voice data into text data with high accuracy.

[0901] Summarization: To summarize the generated text data, we use a pipeline using NLP techniques (e.g., Hugging Face Transformers).

[0902] Cloud storage means: Use cloud storage services (e.g., AWS S3) to store summary data.

[0903] Real-time feedback: Develop a custom application to display analysis results in real time on employees' smart glasses.

[0904] System operation explanation

[0905] Audio data collection

[0906] The server receives and collects the voice data sent from the voice recognition device. For example, when a store employee talks to a customer, the microphone built into the smart glasses captures the voice data in real time.

[0907] Audio data preprocessing

[0908] The server removes noise from the received audio data and normalizes the volume, making the data suitable for analysis.

[0909] Speech recognition and transcription

[0910] The server uses the generative AI model to convert the pre-processed speech data into text data. For example, if a customer asks, "Tell me about next week's campaign," the speech is instantly converted into text.

[0911] Summarizing text data

[0912] The server analyzes the text data, extracts important information, and summarizes it. This allows for efficient organization of key points from long conversations. For example, a summary might be generated that reads, "We need to explain the campaign."

[0913] Data storage and sharing

[0914] The server stores the summary data in cloud storage and makes it accessible to store managers as needed.

[0915] Real-time feedback

[0916] The analysis results are displayed in real time on store employees' smart glasses, enabling them to immediately respond appropriately to customer questions.

[0917] Usage examples and prompt statements

[0918] For example, if a customer asks a store employee, "Tell me about next week's campaign," the smart glasses collect this question as voice data and send it to the server. The server preprocesses the voice data and converts it into text data using a generative AI model. The text data is summarized and the message "Please explain the campaign" is displayed on the smart glasses in real time.

[0919] Example prompt sentence:

[0920] "You will create a program that will analyze and summarize customer conversations in real time and display them on smart glasses. The hardware used will be smart glasses, and a generative AI model will be used for speech recognition. The analysis results will be stored in cloud storage."

[0921] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0922] Step 1:

[0923] The user puts on the smart glasses and starts talking to the customer. The microphone built into the smart glasses captures voice data in real time and sends it to the terminal. The input is the conversation between the customer and the user, and the output is the captured raw voice data.

[0924] Step 2:

[0925] The device sends the captured audio data to the server. The input is the raw audio data sent from the device, and the output is the audio data received by the server.

[0926] Step 3:

[0927] The server preprocesses the audio data it receives, removing noise and normalizing the volume to prepare the audio data for analysis. The input is raw audio data, and the output is clear, preprocessed audio data.

[0928] Step 4:

[0929] The server inputs the preprocessed audio data into a generative AI model and converts the audio data into text data. A generative AI model (e.g., OpenAI Whisper) is used to perform highly accurate transcription. The input is the preprocessed audio data, and the output is the generated text data.

[0930] Step 5:

[0931] The server analyzes the generated text data and summarizes it, extracting important information and summarizing the main points in a compact format. The input is text data, and the output is summarized information.

[0932] Step 6:

[0933] The server saves the summarized information in cloud storage. Specifically, it uploads the data to AWS S3, for example. The input is the summarized information, and the output is the data saved in cloud storage.

[0934] Step 7:

[0935] The analysis results are displayed in real time on the user's smart glasses. As the user continues to interact with the customer, summarized information is displayed on the smart glasses' display. The input is the summarized information, and the output is the feedback displayed on the smart glasses' display.

[0936] Step 8:

[0937] The user responds appropriately to customers based on the feedback from the smart glasses. The input is the information displayed on the smart glasses, and the output is to improve the quality of customer service.

[0938] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0939] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[0940] Audio data collection

[0941] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's have our next meeting on Tuesday next week," this voice is immediately collected by the device.

[0942] Audio data preprocessing

[0943] Once the server receives the collected audio data, it performs noise reduction and volume normalization to prepare the audio data for analysis. For example, background noise and echoes are removed to ensure the audio data is clear.

[0944] Speech recognition and transcription

[0945] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language. For example, a user's statement, "Let's have our next meeting on Tuesday of next week," is accurately output as text data, "Let's have our next meeting on Tuesday of next week."

[0946] Summarizing text data

[0947] The server analyzes the generated text data and extracts and summarizes important information. This allows for efficient organization of key points and important remarks from the meeting. For example, a long text data of the entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[0948] Emotion recognition by emotion engine

[0949] The server uses an emotion engine to recognize the user's emotions from the voice data. For example, it can determine emotional states such as "joy," "anger," or "sadness" from the tone and tempo of the user's speech. The recognized emotions are tagged in the text data and saved as emotional information.

[0950] Data storage and sharing

[0951] The server stores the summarized data and emotion information in cloud storage, and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[0952] Real-time feedback

[0953] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0954] Voice analysis device integration

[0955] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[0956] As described above, the voice analysis system of the present invention can consistently perform processes from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, thereby achieving comprehensive analysis that also includes the user's emotions.

[0957] The processing flow will be explained below.

[0958] Step 1:

[0959] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's schedule our next meeting for next Tuesday," the audio is captured by the device.

[0960] Step 2:

[0961] The terminal periodically transmits the buffered voice data to the server, which stores the received voice data in a database and awaits the next processing step.

[0962] Step 3:

[0963] The server performs noise reduction and volume normalization on the audio data through a pre-processing module, which removes background noise and normalizes the volume to a certain level. For example, audio data with a lot of echo or noise can be denoised to produce clear audio data.

[0964] Step 4:

[0965] The server divides the preprocessed speech data into chunks and inputs them into a generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[0966] Step 5:

[0967] The server analyzes the generated text data, extracts and summarizes important information, for example, summarizing a long meeting as "The next meeting will be held on Tuesday next week."

[0968] Step 6:

[0969] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the voice tone, speed, pitch, etc. to identify emotions such as "joy," "anger," or "sadness," and tags them in the text data. For example, if the user speaks with a happy expression, the tag "joy" is added.

[0970] Step 7:

[0971] The server stores the summarized text data and emotion information in cloud storage, where the data is securely stored and can be accessed by the user later.

[0972] Step 8:

[0973] After the meeting, users can access a web portal to download or share the generated minutes and summary data from the cloud storage. For example, they can share emotion-tagged minutes with team members.

[0974] Step 9:

[0975] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[0976] Step 10:

[0977] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations. This will enable efficient analysis and storage of voice data in the field. For example, a dedicated voice analysis device will be used in a sales situation, and the content of the sales negotiations and emotional information at the time will be converted into text and stored in real time.

[0978] The above steps create a system that can consistently collect, analyze, convert, and share voice data.

[0979] Example 2

[0980] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] Conventional speech analysis systems have issues with noise reduction, volume normalization, data conversion, and summarization in the collection and analysis of speech data, and are particularly lacking in real-time feedback and emotion recognition. Furthermore, collecting and processing speech data in offline environments is difficult, resulting in insufficient user convenience.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0983] In this invention, the server includes means for collecting audio data, means for preprocessing the collected audio data to perform noise reduction and volume normalization, means for converting the preprocessed audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the audio data and tagging the generated text data with the recognized emotions, means for storing the summarized data and emotional information in cloud storage, means for allowing users to access the stored data via a web portal, and means for displaying real-time feedback. This enables a consistent process from audio data collection to analysis, data conversion, emotion recognition, storage, sharing, and real-time feedback.

[0984] "Means for collecting voice data" refers to a function that captures the voice spoken by a user during a meeting or face-to-face conversation using a voice input device such as a microphone on the device, and temporarily stores that voice.

[0985] "Noise removal" is a process that removes unnecessary sounds such as background noise and echo from collected voice data, making it suitable for voice analysis.

[0986] "Volume normalization" is a process that adjusts the volume of collected audio data to a constant level in order to improve the accuracy of analysis and text conversion.

[0987] A "generative AI model" is an artificial intelligence algorithm or program that has the ability to convert voice data into text data, and is specifically designed to achieve highly accurate voice recognition.

[0988] "Means for converting into text data" refers to a function that converts preprocessed audio data into text format data using a generative AI model.

[0989] The "means for summarizing text data" is a function that extracts important information from the generated text data and summarizes the main points in a concise manner.

[0990] The "means for recognizing emotions" is a function that analyzes the user's emotions from voice data, identifies those emotions, and generates appropriate emotion tags.

[0991] The "means for tagging the generated text data" is a function that adds the recognized emotion to the text data as a tag, making the text data easier to search and reference later.

[0992] "Means for saving to cloud storage" is a function that saves summarized text data and its emotion tags on an external cloud server, making it easy to store and share data.

[0993] "Means for allowing users to access via a web portal" means a web-based interface that allows users to access data stored in cloud storage via the Internet.

[0994] The "means for displaying real-time feedback" is a function that displays information such as analyzed text data, its summary, and emotion tags on the user's device in real time.

[0995] The "means for dividing into chunks" is a function for dividing preprocessed audio data into small data blocks at regular intervals for efficient analysis and processing.

[0996] "Means for linking with voice analysis devices" refers to a mechanism that connects to dedicated voice analysis equipment used in face-to-face meetings and sales situations, and provides the function of collecting and analyzing data.

[0997] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[0998] Audio data collection

[0999] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through the microphone and temporarily stores it in a buffer. Specifically, a built-in microphone or an external microphone device can be used. For example, if a user says, "Let's have our next meeting on Tuesday next week," this audio is immediately collected by the device.

[1000] Audio data preprocessing

[1001] Once the server receives the collected audio data, it performs noise reduction and volume normalization using advanced noise filtering algorithms and volume adjustment software, for example, removing background noise and echo to ensure clear audio data.

[1002] Speech recognition and transcription

[1003] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences in the Japanese language and the diversity of pronunciation. For example, a user's statement, "Let's have our next meeting on Tuesday next week," is accurately output as text data as, "Let's have our next meeting on Tuesday next week."

[1004] Summarizing text data

[1005] The server analyzes the generated text data, extracts and summarizes important information, typically using natural language processing algorithms. For example, a long text data entry for an entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[1006] Emotion recognition by emotion engine

[1007] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes parameters such as the tone and tempo of the voice to determine emotions such as "happiness," "anger," and "sadness." For example, "happiness" is recognized from the tone and tempo of the voice when the user speaks, and an emotion tag such as "happiness" is added to the text data.

[1008] Data storage and sharing

[1009] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The cloud storage service can use any popular online storage solution. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[1010] Real-time feedback

[1011] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[1012] Voice analysis device integration

[1013] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[1014] As described above, the voice analysis system of the present invention can consistently perform everything from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, and realizes comprehensive analysis including the user's emotions. This allows users to obtain more accurate and useful information in meetings and face-to-face communications.

[1015] Specific examples

[1016] Example prompt: "Let's schedule our next meeting for next Tuesday." Explain how this system will collect and analyze data and provide feedback to the user.

[1017] Generated description: If a user says, "Let's schedule our next meeting for next Tuesday," the device collects the audio data, and the server performs noise reduction and volume normalization. The generative AI model then converts the audio into text, and the text data is summarized as "Let's schedule our next meeting for next Tuesday." The emotion engine then recognizes this as "joy," and when the user logs in to the portal, they can view the emotion-tagged minutes.

[1018] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1019] Program processing steps

[1020] Step 1: Collecting audio data

[1021] A user participates in a web conferencing system or a face-to-face meeting. The device captures voice data in real time through the microphone and temporarily stores it in a buffer. The input is the user's voice, and the output is the voice data temporarily stored in the buffer. For example, when a user says, "Let's have our next meeting on Tuesday next week," the device's microphone captures the voice, and the data is temporarily stored on the device.

[1022] Step 2: Preprocessing the audio data

[1023] The server receives audio data from the device. It performs noise removal and volume normalization on the received audio data. The input is the audio data stored in the buffer, and the output is the preprocessed audio data. Specifically, background noise and echo are removed from the audio data, and the volume is adjusted to a constant level.

[1024] Step 3: Speech recognition and transcription

[1025] The server inputs preprocessed voice data into the generative AI model. The generative AI model converts the voice data into text data. The input is preprocessed voice data, and the output is text data. Specifically, the voice saying "Let's have our next meeting on Tuesday next week" is converted into text by the generative AI model as "Let's have our next meeting on Tuesday next week."

[1026] Step 4: Summarizing the text data

[1027] The server analyzes the generated text data and extracts and summarizes important information. The input is the generated text data, and the output is the summarized text data. Specifically, the text data is analyzed by a natural language processing algorithm and summarized as "The next meeting will be held on Tuesday next week."

[1028] Step 5: Emotion Recognition with the Emotion Engine

[1029] The server uses an emotion engine to recognize the user's emotion from the voice data. The recognized emotion is then tagged to the text data. The input is voice data and text data, and the output is text data with an emotion tag. Specifically, "joy" is recognized from the voice tone and tempo, and a "joy" tag is added to the text data.

[1030] Step 6: Store and share data

[1031] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The input is emotion-tagged text data, and the output is data stored in cloud storage. Specifically, users can log in to the portal site after the meeting and download the emotion-tagged minutes.

[1032] Step 7: Real-time feedback

[1033] The device displays the progress of transcription and summarization in real time during the meeting. Emotional information recognized by the emotion engine is also displayed in real time. The input is text data and emotional information, and the output is real-time display information that can be checked by the user. Specifically, during the meeting, the message "Let's have our next meeting on Tuesday next week" is displayed, and the "joy" tag is also displayed at the same time.

[1034] Step 8: Linking your voice analysis device

[1035] The terminal works in conjunction with a voice analysis device that is also used in face-to-face meetings and sales situations. This makes it possible to collect and analyze voice data even in offline environments. The input is offline voice data, and the output is text-translated and emotion-tagged data. Specifically, the user uses a dedicated voice analysis device at the sales site, and the content of the sales negotiations and emotional information held there are converted into text in real time and saved.

[1036] (Application example 2)

[1037] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1038] Conventional voice analysis systems were capable of transcribing general voice data and removing noise, but were unable to recognize the speaker's emotional state in real time and provide feedback based on that. This made it difficult for staff to understand the customer's emotions in customer service or sales situations, making it difficult to respond quickly and appropriately. The objective of the present invention is to solve these problems and provide a system that can understand the content and emotions of customers in real time and support appropriate responses.

[1039] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the voice data using an emotion engine, means for saving the summarized data and emotion information in cloud storage, means for displaying real-time feedback, and means for displaying feedback in real time using smart glasses to instantly grasp the customer's comments and emotions. This allows staff to grasp the content of customer comments and emotions in real time and respond promptly and appropriately.

[1040] "Audio data" refers to data that represents an audio signal in digital form.

[1041] "Means for collecting" refers to a method or device for capturing audio data using a microphone or other input device.

[1042] "Preprocessing" refers to the process of removing noise from audio data and normalizing the volume for high-quality analysis.

[1043] "Noise reduction" is a process for removing unnecessary background sounds from audio data to obtain a clear audio signal.

[1044] "Volume normalization" is a process for making the volume level of audio data constant.

[1045] A "generative AI model" is an algorithm or system that uses machine learning technology to analyze voice data with high accuracy.

[1046] "Text data" refers to audio data that has been converted into text and is displayed in an easy-to-read format.

[1047] A "summarizing means" is a method or device for extracting important information from the generated text data and summarizing it concisely.

[1048] An "emotion engine" is a system that analyzes and recognizes people's emotions from the tone of their voice and the content of their statements.

[1049] "Emotion information" is data about the speaker's emotional state obtained from the voice data analyzed by the emotion engine.

[1050] "Cloud storage" is a collection of remote servers that store and access data over the internet.

[1051] "Real-time feedback" is a function that instantly processes voice data and analysis results and provides information to the user immediately.

[1052] "Smart glasses" are glasses-type devices with display functions that are used by the wearer to obtain visual information.

[1053] The "means for displaying feedback" refers to a method or device for providing analysis results or summary information to a user through an output device such as a display.

[1054] "Customer utterances" are messages or questions expressed by customers through speech.

[1055] "Customer service" refers to a series of activities to provide information and services in response to customer needs.

[1056] This invention is a system that supports customer service in brick-and-mortar stores, collecting, preprocessing, transcribing, summarizing, and recognizing emotions from voice data in real time, allowing staff to instantly understand the content and emotions of customers and respond appropriately.

[1057] Hardware and software used

[1058] Hardware

[1059] Smart glasses: Collects voice data in real time and displays the analysis results on a screen.

[1060] Server: Responsible for preprocessing, transcription, summarization, and emotion recognition of audio data.

[1061] software

[1062] Speech Recognition Library (speech_recognition): Converts voice data into text data.

[1063] Noise Reduction Library (noisereduce): Performs noise reduction and volume normalization.

[1064] Generative AI model (openai API): A generative AI model for summarization.

[1065] Emotion Recognizer: Recognizes emotions from audio data.

[1066] Data processing and calculation

[1067] 1. Collection of audio data

[1068] A microphone built into the smart glasses collects customer speech and records it as audio data, which is then sent to a server in real time.

[1069] 2. Pretreatment

[1070] The server removes noise and normalizes the volume of the received audio data to prepare it for analysis.

[1071] 3. Transcription

[1072] The preprocessed speech data is converted into text data using a speech recognition library. For example, a statement such as "I would like to know if new products will be available next month" is accurately converted into text.

[1073] 4. Summarizing Text Data

[1074] The server uses a generative AI model to extract and summarize important information from the generated text data, for example, "Confirm the arrival schedule of new products."

[1075] 5. Emotion recognition

[1076] An emotion recognition engine is used on text data to recognize customer emotions. For example, the emotion "interest" is recognized.

[1077] 6. Real-time feedback

[1078] The smart glasses display summary and sentiment information in real time, allowing staff to instantly understand customer needs and emotions.

[1079] Specific examples

[1080] For example, if a customer says, "I'd like to know if new products will be arriving next month," this speech is collected by the smart glasses' microphone. The audio data is sent to a server where noise is removed and volume normalized. It is then transcribed using a speech recognition library, generating text data such as "I'd like to know if new products will be arriving next month." This text data is then summarized using a generative AI model as "Check the new product arrival schedule," and the emotion recognition engine recognizes the emotion "interest." Finally, this information is displayed in real time on the smart glasses' display, allowing staff to quickly respond to the customer.

[1081] Prompt Sentence Examples

[1082] An example prompt for a generative AI model is:

[1083] Summarize the full text below:

[1084] I would like to know if any new products will be in stock next month.

[1085] summary:

[1086] The above is an embodiment of the present invention, which allows staff to grasp the content of customer comments and emotions in real time, enabling them to respond quickly and appropriately.

[1087] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1088] Step 1:

[1089] Audio data collection

[1090] Input: User (customer) utterance

[1091] Specific operation: The user speaks through the microphone built into the smart glasses. The smart glasses capture the user's speech as audio data in real time and temporarily store the data in the smart glasses.

[1092] Output: Recorded audio data

[1093] Step 2:

[1094] Audio data preprocessing

[1095] Input: Recorded audio data

[1096] How it works: Smart glasses collect and send audio data to a server, which then performs noise reduction and volume normalization on the received audio data to remove background noise and echoes and maintain a consistent volume level.

[1097] Output: Pre-processed clear audio data

[1098] Step 3:

[1099] Transcription

[1100] Input: Preprocessed, clear audio data

[1101] Specific operation: The server converts the preprocessed audio data into text data using a speech recognition library (speech_recognition), which analyzes the audio signal and generates a corresponding string of characters.

[1102] Output: Generated text data (e.g., "I would like to know if new products will be available next month.")

[1103] Step 4:

[1104] Summarizing text data

[1105] Input: Generated text data

[1106] Specific operation: The server uses a generative AI model (openai API) to extract and summarize important information from text data. The generative AI model receives a specific prompt (e.g., "Please summarize the following sentence: I would like to know if new products will be available next month.") and generates a summarized text.

[1107] Output: Summarized text data (e.g., "Confirm new product arrival schedule")

[1108] Step 5:

[1109] emotion recognition

[1110] Input: Generated text data

[1111] Specific operation: The server uses an emotion recognition engine (EmotionRecognizer) to extract emotional information from text data. The emotion recognition engine analyzes the emotional elements contained in the text and recognizes the emotional state (e.g., interest).

[1112] Output: Recognized emotion information (e.g., "interest")

[1113] Step 6:

[1114] View real-time feedback

[1115] Input: Summarized text data and recognized emotion information

[1116] How it works: The server sends summarized text data and emotional information to the smart glasses, which then display it in real time on the display for the user to check immediately.

[1117] Output: Real-time summary information and emotion information (e.g., "Check new product arrival schedule" and "Interested" are displayed on the screen)

[1118] Step 7:

[1119] Data storage and sharing

[1120] Input: Summarized text data and sentiment information

[1121] How it works: The server stores the summarized text data and emotion information in cloud storage, which can be accessed later and downloaded by users via a web portal to share with team members as needed.

[1122] Output: Text data and emotion information stored in cloud storage

[1123] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1124] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1125] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1126] [Fourth embodiment]

[1127] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1128] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1129] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1130] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1131] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1132] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1133] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1134] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1135] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1136] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1137] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1138] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1139] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1140] This invention is a speech analysis system that collects speech data and analyzes and summarizes it with high accuracy. Each component cooperates to achieve efficient and accurate speech analysis. Below, we will explain specific examples of processing at each stage of this system.

[1141] Audio data collection

[1142] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through a microphone.

[1143] For example, if a user says, "Let's have our next meeting on Tuesday next week," this speech is immediately collected by the terminal.

[1144] Audio data preprocessing

[1145] Once the server receives the collected audio data, it performs noise reduction and volume normalization, making the audio data suitable for analysis.

[1146] For example, if background noise or echo is present, it will be removed, leaving the audio data clear.

[1147] Speech recognition and transcription

[1148] The server uses a generative AI model to convert the preprocessed audio data into text data, which then generates highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language.

[1149] For example, a user's statement "Let's have the next meeting on Tuesday next week" is accurately output as text data as "Let's have the next meeting on Tuesday next week."

[1150] Summarizing text data

[1151] The server analyzes the generated text data, extracts and summarizes important information, and efficiently organizes the main points and important comments from the meeting.

[1152] For example, a long text data of an entire meeting can be summarized as "The next meeting will be held next Tuesday."

[1153] Data storage and sharing

[1154] The server stores the summarized data in cloud storage and makes it accessible to users through a web portal, allowing them to easily view and share the data they need.

[1155] For example, after a meeting, a user can log in to a portal site, download the minutes, and share them with team members.

[1156] Real-time feedback

[1157] The device displays real-time transcription and analysis results, allowing users to keep track of progress during the meeting and receive feedback as needed.

[1158] For example, important statements made during a meeting are converted into text in real time and displayed on the screen.

[1159] Voice analysis device integration

[1160] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations, making voice analysis possible even in offline environments, and allowing for efficient recording.

[1161] For example, a user can use a dedicated voice analysis device at a sales site, and the contents of business negotiations conducted there will be converted into text in real time and saved.

[1162] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, helping to improve the efficiency of communication both inside and outside the company.

[1163] The processing flow will be explained below.

[1164] Step 1:

[1165] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer.

[1166] Step 2:

[1167] The device periodically sends the collected voice data to a server, allowing the data to be analyzed in real time.

[1168] Step 3:

[1169] The server passes the received audio data to a pre-processing module for noise reduction and volume normalization. Noise reduction removes background sounds and background noise, while volume normalization maintains consistent audio quality.

[1170] Step 4:

[1171] The server inputs the preprocessed speech data into the generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[1172] Step 5:

[1173] The server analyzes the generated text data and extracts important keywords and phrases, allowing for an efficient and concise summary of the meeting content.

[1174] Step 6:

[1175] The server stores the summarized text data in cloud storage, and the summarized data containing important information is safely stored in the storage.

[1176] Step 7:

[1177] After the meeting, users can access the web portal and download or share the minutes and summary data generated from the cloud storage, making it easier to share information within the company.

[1178] Step 8:

[1179] The device displays the progress of transcription and summarization in real time during the meeting, allowing users to check what is being said in real time and make corrections or additions on the spot as needed.

[1180] Step 9:

[1181] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations, making it possible to efficiently analyze and store voice data on-site.

[1182] The above steps create a system that consistently collects, analyzes, stores, and shares voice data.

[1183] Example 1

[1184] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1185] Conventional voice collection and analysis systems have limitations in the accuracy of real-time transcription and analysis, and suffer from insufficient noise reduction and volume normalization. They also face problems with the efficiency of analyzing and summarizing collected voice data. Furthermore, device integration is insufficient when used in face-to-face meetings or sales situations, making voice analysis in offline environments difficult.

[1186] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1187] In this invention, the server includes means for collecting audio data, means for performing preprocessing, noise reduction, and volume normalization, means for converting the collected audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in remote storage, means for displaying real-time feedback, and means for analyzing and processing the audio data. This enables highly accurate audio analysis and summarization in real time, allowing for efficient and effective data collection, processing, and sharing.

[1188] "Voice data" means data that is a digital representation of a user's voice.

[1189] "Preprocessing" refers to a series of processes for preparing collected audio data to make it easier to analyze, and includes noise removal and volume normalization.

[1190] "Noise reduction" is a process that removes unnecessary background sounds and noise from audio data.

[1191] "Volume normalization" is a process for making the volume level of audio data uniform.

[1192] A "generative AI model" refers to an algorithm or statistical model that uses artificial intelligence to perform a specific task and is used to convert voice data into text.

[1193] "Text data" is data obtained by converting voice data into a character string.

[1194] "Summarization" refers to concisely summarizing information and includes the process of extracting important information.

[1195] "Remote storage" is a data storage area that can be accessed via a network, such as a cloud environment.

[1196] "Real-time feedback" refers to the function of returning processing results to the user immediately.

[1197] A "voice analysis device" is a device or system dedicated to analyzing voice data.

[1198] The present invention is a voice analysis system in which each component cooperates to achieve efficient and accurate voice analysis. A specific method for implementing this system will be described in detail below.

[1199] Audio data collection

[1200] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time using the device's built-in microphone. The device is often a PC or smartphone. For example, if a user says, "Let's schedule our next meeting for next Tuesday," this audio is captured by the device's microphone.

[1201] Audio data preprocessing

[1202] When audio data is sent from the device to the server, the server uses Python libraries (such as Librosa or Pydub) to remove noise and normalize the volume. This process prepares the audio data for analysis. For example, if background noise or echo is present, it is removed, resulting in clear audio data.

[1203] Speech recognition and transcription

[1204] The server that receives the preprocessed voice data converts it into text data using a generative AI model such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text. This conversion accurately converts the speech into text. For example, a user uttering "Let's have our next meeting on Tuesday next week" is accurately output as text data as "Let's have our next meeting on Tuesday next week."

[1205] Summarizing text data

[1206] The server then analyzes the converted text data using a natural language processing (NLP) library (e.g., Spacy or NLTK) to extract and summarize important information. This allows for efficient extraction of key points from large amounts of text data. For example, a long meeting's minutes can be summarized as "The next meeting will be held next Tuesday."

[1207] Data storage and sharing

[1208] The server stores the summarized data in a remote storage (e.g., AWS S3, Google Cloud Storage), and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the generated minutes, and share them with team members.

[1209] Real-time feedback

[1210] The device displays the transcription and analysis results in real time, allowing users to understand the progress of the meeting and receive immediate feedback as needed. For example, anything said during a meeting is immediately transcribed and displayed on the device screen.

[1211] Linking voice analysis devices

[1212] The device works in conjunction with a voice analysis device used in face-to-face meetings and sales situations. This allows for voice analysis even in offline environments, making record-keeping more efficient. For example, a user can use a dedicated voice analysis device at a sales site to convert the contents of a sales negotiation into text in real time and save it.

[1213] Examples of prompt statements

[1214] Below are some specific examples of prompt sentences to be input to the generative AI model and the results of their processing.

[1215] Prompt statement:

[1216] Please transcribe what is said and then summarize it. Here is the audio: "Let's have our next meeting next Tuesday."

[1217] Processing result:

[1218] Transcription result: Let's have our next meeting next Tuesday.

[1219] Summary result: The next meeting will be next Tuesday.

[1220] In this way, the processes at each stage work together to achieve efficient and highly accurate voice analysis, making it possible to significantly improve the efficiency of communication both inside and outside a company.

[1221] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1222] Step 1:

[1223] Audio data collection

[1224] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time using the device's built-in microphone. The input is the user's voice obtained through the microphone, and the output is the captured voice data. For example, a user might say, "Let's have our next meeting on Tuesday next week," and this is captured by the device's microphone.

[1225] Step 2:

[1226] Sending audio data to the server

[1227] The terminal sends the captured voice data to the server. The input is the captured voice data obtained in step 1, and the output is the voice data sent to the server. Specifically, the voice data is compressed and converted into packets, and then sent to the server.

[1228] Step 3:

[1229] Audio data preprocessing

[1230] The server uses Python libraries (e.g., Librosa or Pydub) to remove noise and normalize the volume of the audio data it receives. The input is the raw audio data received by the server, and the output is clear audio data that has been preprocessed. Specifically, the Librosa library is used to filter background noise, and the Pydub library is used to equalize the volume level.

[1231] Step 4:

[1232] Transcription of audio data

[1233] The server uses a generative AI model, such as Google Cloud Speech-to-Text API or IBM Watson Speech to Text, to convert the preprocessed audio data into text. The input is clear preprocessed audio data, and the output is text data that details the audio content. Specific operations include sending audio data to the API and receiving a text response.

[1234] Step 5:

[1235] Summarizing text data

[1236] The server analyzes the converted text data using a natural language processing (NLP) library (such as Spacy or NLTK) to extract and summarize important information. The input is the full text data converted from speech, and the output is a summary text that summarizes the important points. Specifically, the Spacy library is used to analyze the content of the text data and extract important phrases and keywords.

[1237] Step 6:

[1238] Data storage and sharing

[1239] The server stores the summarized data in remote storage (e.g., AWS S3 or Google Cloud Storage) and makes it accessible to users through a web portal. The input is the summarized text data, and the output is the data stored in cloud storage. Specifically, a URL and access key for accessing the stored data are generated and linked to the user's account.

[1240] Step 7:

[1241] Providing real-time feedback

[1242] The device displays the transcription and analysis results in real time. The input is the transcription and summary data sent sequentially from the server, and the output is feedback information displayed on the user's screen. Specifically, the user's remarks are transcribed in real time, and the device screen displays, "Let's schedule the next meeting for next Tuesday."

[1243] Step 8:

[1244] Linking voice analysis devices

[1245] The device connects to a voice analysis device used in face-to-face meetings and sales situations. The input is voice data obtained from a dedicated voice analysis device, and the output is text data as the analysis results. Specific operations include analyzing voice data collected offline in real time and saving it on the device.

[1246] (Application example 1)

[1247] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1248] Conventional speech analysis systems have limitations in accuracy and efficiency at each step of speech data collection, preprocessing, transcription, and summarization, making it difficult to provide real-time feedback or effectively utilize analysis results, especially in dynamic environments such as stores. This creates problems such as poor customer service quality and limited employee training effectiveness.

[1249] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1250] In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for storing the summarized data in cloud storage, means for displaying real-time feedback, means for analyzing the voice data sent from the voice recognition device, and means for store employees to receive the analysis results in real time. This enables high-precision voice analysis and immediate feedback even in a dynamic store environment, thereby improving the quality of customer service and maximizing the effectiveness of employee training.

[1251] "Audio data" refers to data that is collected and stored in digital form.

[1252] "Preprocessing" refers to the process of removing noise from collected audio data and normalizing the volume to make it suitable for analysis.

[1253] A "generative AI model" is an artificial intelligence model used to convert voice data into text data.

[1254] "Text data" refers to data obtained by converting voice data into text format.

[1255] "Summarization" is the process of extracting important information from the generated text data and summarizing it in a compact format.

[1256] "Cloud storage" is a remote server for storing and managing data over the Internet.

[1257] "Real-time feedback" is a function that instantly conveys the results of voice analysis to users on the spot.

[1258] A "voice recognition device" is a hardware device for collecting and analyzing speech.

[1259] "Analysis" is the process of understanding the content and extracting meaning from collected data.

[1260] "Store employees" are personnel who deal with customers and perform other duties in a store.

[1261] The present invention is a system that collects and analyzes voice data in real time and provides immediate feedback to improve the quality of customer service in physical stores. The following describes an embodiment of the present invention.

[1262] Program Overview

[1263] The server, voice recognition device, and store employee terminals (such as smart glasses) work together.

[1264] Hardware and Software Configuration

[1265] The server is configured using the following software and hardware:

[1266] Voice data collection method: Smart glasses with built-in microphones and portable voice recorders are used as voice recognition devices.

[1267] Preprocessing method: To preprocess the audio data, software for noise reduction and volume normalization (e.g., Audacity) is used.

[1268] Generative AI models: Use generative AI models (e.g., OpenAI Whisper) to convert voice data into text data with high accuracy.

[1269] Summarization: To summarize the generated text data, we use a pipeline using NLP techniques (e.g., Hugging Face Transformers).

[1270] Cloud storage means: Use cloud storage services (e.g., AWS S3) to store summary data.

[1271] Real-time feedback: Develop a custom application to display analysis results in real time on employees' smart glasses.

[1272] System operation explanation

[1273] Audio data collection

[1274] The server receives and collects the voice data sent from the voice recognition device. For example, when a store employee talks to a customer, the microphone built into the smart glasses captures the voice data in real time.

[1275] Audio data preprocessing

[1276] The server removes noise from the received audio data and normalizes the volume, making the data suitable for analysis.

[1277] Speech recognition and transcription

[1278] The server uses the generative AI model to convert the pre-processed speech data into text data. For example, if a customer asks, "Tell me about next week's campaign," the speech is instantly converted into text.

[1279] Summarizing text data

[1280] The server analyzes the text data, extracts important information, and summarizes it. This allows for efficient organization of key points from long conversations. For example, a summary might be generated that reads, "We need to explain the campaign."

[1281] Data storage and sharing

[1282] The server stores the summary data in cloud storage and makes it accessible to store managers as needed.

[1283] Real-time feedback

[1284] The analysis results are displayed in real time on store employees' smart glasses, enabling them to immediately respond appropriately to customer questions.

[1285] Usage examples and prompt statements

[1286] For example, if a customer asks a store employee, "Tell me about next week's campaign," the smart glasses collect this question as voice data and send it to the server. The server preprocesses the voice data and converts it into text data using a generative AI model. The text data is summarized and the message "Please explain the campaign" is displayed on the smart glasses in real time.

[1287] Example prompt sentence:

[1288] "You will create a program that will analyze and summarize customer conversations in real time and display them on smart glasses. The hardware used will be smart glasses, and a generative AI model will be used for speech recognition. The analysis results will be stored in cloud storage."

[1289] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1290] Step 1:

[1291] The user puts on the smart glasses and starts talking to the customer. The microphone built into the smart glasses captures voice data in real time and sends it to the terminal. The input is the conversation between the customer and the user, and the output is the captured raw voice data.

[1292] Step 2:

[1293] The device sends the captured audio data to the server. The input is the raw audio data sent from the device, and the output is the audio data received by the server.

[1294] Step 3:

[1295] The server preprocesses the audio data it receives, removing noise and normalizing the volume to prepare the audio data for analysis. The input is raw audio data, and the output is clear, preprocessed audio data.

[1296] Step 4:

[1297] The server inputs the preprocessed audio data into a generative AI model and converts the audio data into text data. A generative AI model (e.g., OpenAI Whisper) is used to perform highly accurate transcription. The input is the preprocessed audio data, and the output is the generated text data.

[1298] Step 5:

[1299] The server analyzes the generated text data and summarizes it, extracting important information and summarizing the main points in a compact format. The input is text data, and the output is summarized information.

[1300] Step 6:

[1301] The server saves the summarized information in cloud storage. Specifically, it uploads the data to AWS S3, for example. The input is the summarized information, and the output is the data saved in cloud storage.

[1302] Step 7:

[1303] The analysis results are displayed in real time on the user's smart glasses. As the user continues to interact with the customer, summarized information is displayed on the smart glasses' display. The input is the summarized information, and the output is the feedback displayed on the smart glasses' display.

[1304] Step 8:

[1305] The user responds appropriately to customers based on the feedback from the smart glasses. The input is the information displayed on the smart glasses, and the output is to improve the quality of customer service.

[1306] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1307] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[1308] Audio data collection

[1309] When a user participates in a web conferencing system or a face-to-face meeting, the device captures voice data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's have our next meeting on Tuesday next week," this voice is immediately collected by the device.

[1310] Audio data preprocessing

[1311] Once the server receives the collected audio data, it performs noise reduction and volume normalization to prepare the audio data for analysis. For example, background noise and echoes are removed to ensure the audio data is clear.

[1312] Speech recognition and transcription

[1313] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences and idiosyncrasies of the Japanese language. For example, a user's statement, "Let's have our next meeting on Tuesday of next week," is accurately output as text data, "Let's have our next meeting on Tuesday of next week."

[1314] Summarizing text data

[1315] The server analyzes the generated text data and extracts and summarizes important information. This allows for efficient organization of key points and important remarks from the meeting. For example, a long text data of the entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[1316] Emotion recognition by emotion engine

[1317] The server uses an emotion engine to recognize the user's emotions from the voice data. For example, it can determine emotional states such as "joy," "anger," or "sadness" from the tone and tempo of the user's speech. The recognized emotions are tagged in the text data and saved as emotional information.

[1318] Data storage and sharing

[1319] The server stores the summarized data and emotion information in cloud storage, and users can access it through a web portal. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[1320] Real-time feedback

[1321] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[1322] Voice analysis device integration

[1323] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[1324] As described above, the voice analysis system of the present invention can consistently perform processes from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, thereby achieving comprehensive analysis that also includes the user's emotions.

[1325] The processing flow will be explained below.

[1326] Step 1:

[1327] A user starts a web conferencing system or a face-to-face meeting. The device captures audio data in real time through the microphone and temporarily stores it in a buffer. For example, if a user says, "Let's schedule our next meeting for next Tuesday," the audio is captured by the device.

[1328] Step 2:

[1329] The terminal periodically transmits the buffered voice data to the server, which stores the received voice data in a database and awaits the next processing step.

[1330] Step 3:

[1331] The server performs noise reduction and volume normalization on the audio data through a pre-processing module, which removes background noise and normalizes the volume to a certain level. For example, audio data with a lot of echo or noise can be denoised to produce clear audio data.

[1332] Step 4:

[1333] The server divides the preprocessed speech data into chunks and inputs them into a generative AI model, which then analyzes the speech data and outputs the corresponding text data. For example, a speech that says, "Let's have our next meeting on Tuesday next week" is converted into accurate text.

[1334] Step 5:

[1335] The server analyzes the generated text data, extracts and summarizes important information, for example, summarizing a long meeting as "The next meeting will be held on Tuesday next week."

[1336] Step 6:

[1337] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes the voice tone, speed, pitch, etc. to identify emotions such as "joy," "anger," or "sadness," and tags them in the text data. For example, if the user speaks with a happy expression, the tag "joy" is added.

[1338] Step 7:

[1339] The server stores the summarized text data and emotion information in cloud storage, where the data is securely stored and can be accessed by the user later.

[1340] Step 8:

[1341] After the meeting, users can access a web portal to download or share the generated minutes and summary data from the cloud storage. For example, they can share emotion-tagged minutes with team members.

[1342] Step 9:

[1343] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[1344] Step 10:

[1345] The device will be linked to a voice analysis device so that it can be used in face-to-face meetings and sales situations. This will enable efficient analysis and storage of voice data in the field. For example, a dedicated voice analysis device will be used in a sales situation, and the content of the sales negotiations and emotional information at the time will be converted into text and stored in real time.

[1346] The above steps create a system that can consistently collect, analyze, convert, and share voice data.

[1347] Example 2

[1348] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1349] Conventional speech analysis systems have issues with noise reduction, volume normalization, data conversion, and summarization in the collection and analysis of speech data, and are particularly lacking in real-time feedback and emotion recognition. Furthermore, collecting and processing speech data in offline environments is difficult, resulting in insufficient user convenience.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1351] In this invention, the server includes means for collecting audio data, means for preprocessing the collected audio data to perform noise reduction and volume normalization, means for converting the preprocessed audio data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the audio data and tagging the generated text data with the recognized emotions, means for storing the summarized data and emotional information in cloud storage, means for allowing users to access the stored data via a web portal, and means for displaying real-time feedback. This enables a consistent process from audio data collection to analysis, data conversion, emotion recognition, storage, sharing, and real-time feedback.

[1352] "Means for collecting voice data" refers to a function that captures the voice spoken by a user during a meeting or face-to-face conversation using a voice input device such as a microphone on the device, and temporarily stores that voice.

[1353] "Noise removal" is a process that removes unnecessary sounds such as background noise and echo from collected voice data, making it suitable for voice analysis.

[1354] "Volume normalization" is a process that adjusts the volume of collected audio data to a constant level in order to improve the accuracy of analysis and text conversion.

[1355] A "generative AI model" is an artificial intelligence algorithm or program that has the ability to convert voice data into text data, and is specifically designed to achieve highly accurate voice recognition.

[1356] "Means for converting into text data" refers to a function that converts preprocessed audio data into text format data using a generative AI model.

[1357] The "means for summarizing text data" is a function that extracts important information from the generated text data and summarizes the main points in a concise manner.

[1358] The "means for recognizing emotions" is a function that analyzes the user's emotions from voice data, identifies those emotions, and generates appropriate emotion tags.

[1359] The "means for tagging the generated text data" is a function that adds the recognized emotion to the text data as a tag, making the text data easier to search and reference later.

[1360] "Means for saving to cloud storage" is a function that saves summarized text data and its emotion tags on an external cloud server, making it easy to store and share data.

[1361] "Means for allowing users to access via a web portal" means a web-based interface that allows users to access data stored in cloud storage via the Internet.

[1362] The "means for displaying real-time feedback" is a function that displays information such as analyzed text data, its summary, and emotion tags on the user's device in real time.

[1363] The "means for dividing into chunks" is a function for dividing preprocessed audio data into small data blocks at regular intervals for efficient analysis and processing.

[1364] "Means for linking with voice analysis devices" refers to a mechanism that connects to dedicated voice analysis equipment used in face-to-face meetings and sales situations, and provides the function of collecting and analyzing data.

[1365] This invention is a voice analysis system that collects voice data, analyzes and summarizes it with high accuracy, and combines it with an emotion engine that recognizes the user's emotions. Below, we will explain the processing at each stage of this system with concrete examples.

[1366] Audio data collection

[1367] When a user participates in a web conferencing system or a face-to-face meeting, the device captures audio data in real time through the microphone and temporarily stores it in a buffer. Specifically, a built-in microphone or an external microphone device can be used. For example, if a user says, "Let's have our next meeting on Tuesday next week," this audio is immediately collected by the device.

[1368] Audio data preprocessing

[1369] Once the server receives the collected audio data, it performs noise reduction and volume normalization using advanced noise filtering algorithms and volume adjustment software, for example, removing background noise and echo to ensure clear audio data.

[1370] Speech recognition and transcription

[1371] The server uses a generative AI model to convert the preprocessed speech data into text data. The generative AI model produces highly accurate transcriptions, taking into account regional differences in the Japanese language and the diversity of pronunciation. For example, a user's statement, "Let's have our next meeting on Tuesday next week," is accurately output as text data as, "Let's have our next meeting on Tuesday next week."

[1372] Summarizing text data

[1373] The server analyzes the generated text data, extracts and summarizes important information, typically using natural language processing algorithms. For example, a long text data entry for an entire meeting can be summarized as "The next meeting will be held on Tuesday of next week."

[1374] Emotion recognition by emotion engine

[1375] The server uses an emotion engine to recognize the user's emotions from the voice data. The emotion engine analyzes parameters such as the tone and tempo of the voice to determine emotions such as "happiness," "anger," and "sadness." For example, "happiness" is recognized from the tone and tempo of the voice when the user speaks, and an emotion tag such as "happiness" is added to the text data.

[1376] Data storage and sharing

[1377] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The cloud storage service can use any popular online storage solution. For example, after a meeting, a user can log in to the portal site, download the emotion-tagged minutes, and share them with team members.

[1378] Real-time feedback

[1379] The device displays the progress of transcription and summarization in real time during a meeting. In addition, emotional information recognized by the emotion engine is also displayed in real time. This allows users to instantly understand the content of what is being said and the emotional nuances behind it. For example, when the device displays "Let's have our next meeting on Tuesday next week," it will also display the tag "joy."

[1380] Voice analysis device integration

[1381] The device will be linked to a voice analysis device that is also used in face-to-face meetings and sales situations. This will enable the collection and analysis of voice data even in offline environments. For example, a user can use a dedicated voice analysis device in a sales situation, and the content of the sales negotiation and the emotional information at that time will be converted into text and saved in real time.

[1382] As described above, the voice analysis system of the present invention can consistently perform everything from collecting voice data to analyzing, converting it into text, recognizing emotions, saving, and sharing it, and realizes comprehensive analysis including the user's emotions. This allows users to obtain more accurate and useful information in meetings and face-to-face communications.

[1383] Specific examples

[1384] Example prompt: "Let's schedule our next meeting for next Tuesday." Explain how this system will collect and analyze data and provide feedback to the user.

[1385] Generated description: If a user says, "Let's schedule our next meeting for next Tuesday," the device collects the audio data, and the server performs noise reduction and volume normalization. The generative AI model then converts the audio into text, and the text data is summarized as "Let's schedule our next meeting for next Tuesday." The emotion engine then recognizes this as "joy," and when the user logs in to the portal, they can view the emotion-tagged minutes.

[1386] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1387] Program processing steps

[1388] Step 1: Collecting audio data

[1389] A user participates in a web conferencing system or a face-to-face meeting. The device captures voice data in real time through the microphone and temporarily stores it in a buffer. The input is the user's voice, and the output is the voice data temporarily stored in the buffer. For example, when a user says, "Let's have our next meeting on Tuesday next week," the device's microphone captures the voice, and the data is temporarily stored on the device.

[1390] Step 2: Preprocessing the audio data

[1391] The server receives audio data from the device. It performs noise removal and volume normalization on the received audio data. The input is the audio data stored in the buffer, and the output is the preprocessed audio data. Specifically, background noise and echo are removed from the audio data, and the volume is adjusted to a constant level.

[1392] Step 3: Speech recognition and transcription

[1393] The server inputs preprocessed voice data into the generative AI model. The generative AI model converts the voice data into text data. The input is preprocessed voice data, and the output is text data. Specifically, the voice saying "Let's have our next meeting on Tuesday next week" is converted into text by the generative AI model as "Let's have our next meeting on Tuesday next week."

[1394] Step 4: Summarizing the text data

[1395] The server analyzes the generated text data and extracts and summarizes important information. The input is the generated text data, and the output is the summarized text data. Specifically, the text data is analyzed by a natural language processing algorithm and summarized as "The next meeting will be held on Tuesday next week."

[1396] Step 5: Emotion Recognition with the Emotion Engine

[1397] The server uses an emotion engine to recognize the user's emotion from the voice data. The recognized emotion is then tagged to the text data. The input is voice data and text data, and the output is text data with an emotion tag. Specifically, "joy" is recognized from the voice tone and tempo, and a "joy" tag is added to the text data.

[1398] Step 6: Store and share data

[1399] The server stores the summarized data and emotion information in cloud storage, which users can access through a web portal. The input is emotion-tagged text data, and the output is data stored in cloud storage. Specifically, users can log in to the portal site after the meeting and download the emotion-tagged minutes.

[1400] Step 7: Real-time feedback

[1401] The device displays the progress of transcription and summarization in real time during the meeting. Emotional information recognized by the emotion engine is also displayed in real time. The input is text data and emotional information, and the output is real-time display information that can be checked by the user. Specifically, during the meeting, the message "Let's have our next meeting on Tuesday next week" is displayed, and the "joy" tag is also displayed at the same time.

[1402] Step 8: Linking your voice analysis device

[1403] The terminal works in conjunction with a voice analysis device that is also used in face-to-face meetings and sales situations. This makes it possible to collect and analyze voice data even in offline environments. The input is offline voice data, and the output is text-translated and emotion-tagged data. Specifically, the user uses a dedicated voice analysis device at the sales site, and the content of the sales negotiations and emotional information held there are converted into text in real time and saved.

[1404] (Application example 2)

[1405] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1406] Conventional voice analysis systems were capable of transcribing general voice data and removing noise, but were unable to recognize the speaker's emotional state in real time and provide feedback based on that. This made it difficult for staff to understand the customer's emotions in customer service or sales situations, making it difficult to respond quickly and appropriately. The objective of the present invention is to solve these problems and provide a system that can understand the content and emotions of customers in real time and support appropriate responses.

[1407] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for performing preprocessing to remove noise and normalize volume, means for converting the collected voice data into text data using a generative AI model, means for summarizing the generated text data and extracting important information, means for recognizing emotions from the voice data using an emotion engine, means for saving the summarized data and emotion information in cloud storage, means for displaying real-time feedback, and means for displaying feedback in real time using smart glasses to instantly grasp the customer's comments and emotions. This allows staff to grasp the content of customer comments and emotions in real time and respond promptly and appropriately.

[1408] "Audio data" refers to data that represents an audio signal in digital form.

[1409] "Means for collecting" refers to a method or device for capturing audio data using a microphone or other input device.

[1410] "Preprocessing" refers to the process of removing noise from audio data and normalizing the volume for high-quality analysis.

[1411] "Noise reduction" is a process for removing unnecessary background sounds from audio data to obtain a clear audio signal.

[1412] "Volume normalization" is a process for making the volume level of audio data constant.

[1413] A "generative AI model" is an algorithm or system that uses machine learning technology to analyze voice data with high accuracy.

[1414] "Text data" refers to audio data that has been converted into text and is displayed in an easy-to-read format.

[1415] A "summarizing means" is a method or device for extracting important information from the generated text data and summarizing it concisely.

[1416] An "emotion engine" is a system that analyzes and recognizes people's emotions from the tone of their voice and the content of their statements.

[1417] "Emotion information" is data about the speaker's emotional state obtained from the voice data analyzed by the emotion engine.

[1418] "Cloud storage" is a collection of remote servers that store and access data over the internet.

[1419] "Real-time feedback" is a function that instantly processes voice data and analysis results and provides information to the user immediately.

[1420] "Smart glasses" are glasses-type devices with display functions that are used by the wearer to obtain visual information.

[1421] The "means for displaying feedback" refers to a method or device for providing analysis results or summary information to a user through an output device such as a display.

[1422] "Customer utterances" are messages or questions expressed by customers through speech.

[1423] "Customer service" refers to a series of activities to provide information and services in response to customer needs.

[1424] This invention is a system that supports customer service in brick-and-mortar stores, collecting, preprocessing, transcribing, summarizing, and recognizing emotions from voice data in real time, allowing staff to instantly understand the content and emotions of customers and respond appropriately.

[1425] Hardware and software used

[1426] Hardware

[1427] Smart glasses: Collects voice data in real time and displays the analysis results on a screen.

[1428] Server: Responsible for preprocessing, transcription, summarization, and emotion recognition of audio data.

[1429] software

[1430] Speech Recognition Library (speech_recognition): Converts voice data into text data.

[1431] Noise Reduction Library (noisereduce): Performs noise reduction and volume normalization.

[1432] Generative AI model (openai API): A generative AI model for summarization.

[1433] Emotion Recognizer: Recognizes emotions from audio data.

[1434] Data processing and calculation

[1435] 1. Collection of audio data

[1436] A microphone built into the smart glasses collects customer speech and records it as audio data, which is then sent to a server in real time.

[1437] 2. Pretreatment

[1438] The server removes noise and normalizes the volume of the received audio data to prepare it for analysis.

[1439] 3. Transcription

[1440] The preprocessed speech data is converted into text data using a speech recognition library. For example, a statement such as "I would like to know if new products will be available next month" is accurately converted into text.

[1441] 4. Summarizing Text Data

[1442] The server uses a generative AI model to extract and summarize important information from the generated text data, for example, "Confirm the arrival schedule of new products."

[1443] 5. Emotion recognition

[1444] An emotion recognition engine is used on text data to recognize customer emotions. For example, the emotion "interest" is recognized.

[1445] 6. Real-time feedback

[1446] The smart glasses display summary and sentiment information in real time, allowing staff to instantly understand customer needs and emotions.

[1447] Specific examples

[1448] For example, if a customer says, "I'd like to know if new products will be arriving next month," this speech is collected by the smart glasses' microphone. The audio data is sent to a server where noise is removed and volume normalized. It is then transcribed using a speech recognition library, generating text data such as "I'd like to know if new products will be arriving next month." This text data is then summarized using a generative AI model as "Check the new product arrival schedule," and the emotion recognition engine recognizes the emotion "interest." Finally, this information is displayed in real time on the smart glasses' display, allowing staff to quickly respond to the customer.

[1449] Prompt Sentence Examples

[1450] An example prompt for a generative AI model is:

[1451] Summarize the full text below:

[1452] I would like to know if any new products will be in stock next month.

[1453] summary:

[1454] The above is an embodiment of the present invention, which allows staff to grasp the content of customer comments and emotions in real time, enabling them to respond quickly and appropriately.

[1455] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1456] Step 1:

[1457] Audio data collection

[1458] Input: User (customer) utterance

[1459] Specific operation: The user speaks through the microphone built into the smart glasses. The smart glasses capture the user's speech as audio data in real time and temporarily store the data in the smart glasses.

[1460] Output: Recorded audio data

[1461] Step 2:

[1462] Audio data preprocessing

[1463] Input: Recorded audio data

[1464] How it works: Smart glasses collect and send audio data to a server, which then performs noise reduction and volume normalization on the received audio data to remove background noise and echoes and maintain a consistent volume level.

[1465] Output: Pre-processed clear audio data

[1466] Step 3:

[1467] Transcription

[1468] Input: Preprocessed, clear audio data

[1469] Specific operation: The server converts the preprocessed audio data into text data using a speech recognition library (speech_recognition), which analyzes the audio signal and generates a corresponding string of characters.

[1470] Output: Generated text data (e.g., "I would like to know if new products will be available next month.")

[1471] Step 4:

[1472] Summarizing text data

[1473] Input: Generated text data

[1474] Specific operation: The server uses a generative AI model (openai API) to extract and summarize important information from text data. The generative AI model receives a specific prompt (e.g., "Please summarize the following sentence: I would like to know if new products will be available next month.") and generates a summarized text.

[1475] Output: Summarized text data (e.g., "Confirm new product arrival schedule")

[1476] Step 5:

[1477] emotion recognition

[1478] Input: Generated text data

[1479] Specific operation: The server uses an emotion recognition engine (EmotionRecognizer) to extract emotional information from text data. The emotion recognition engine analyzes the emotional elements contained in the text and recognizes the emotional state (e.g., interest).

[1480] Output: Recognized emotion information (e.g., "interest")

[1481] Step 6:

[1482] View real-time feedback

[1483] Input: Summarized text data and recognized emotion information

[1484] How it works: The server sends summarized text data and emotional information to the smart glasses, which then display it in real time on the display for the user to check immediately.

[1485] Output: Real-time summary information and emotion information (e.g., "Check new product arrival schedule" and "Interested" are displayed on the screen)

[1486] Step 7:

[1487] Data storage and sharing

[1488] Input: Summarized text data and sentiment information

[1489] How it works: The server stores the summarized text data and emotion information in cloud storage, which can be accessed later and downloaded by users via a web portal to share with team members as needed.

[1490] Output: Text data and emotion information stored in cloud storage

[1491] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1492] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1493] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1494] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1495] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1496] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1497] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1498] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1499] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1500] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1501] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1502] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1503] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1504] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1505] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1506] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1507] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1508] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1509] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1510] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1511] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1512] The following is further disclosed regarding the above embodiment.

[1513] (Claim 1)

[1514] means for collecting audio data;

[1515] means for pre-processing to perform noise removal and volume normalization;

[1516] a means for converting the collected voice data into text data using a generative AI model;

[1517] A means for summarizing the generated text data and extracting important information;

[1518] a means for storing the summarized data in cloud storage;

[1519] a means for displaying real-time feedback;

[1520] A system including:

[1521] (Claim 2)

[1522] and means for chunking the preprocessed audio data.

[1523] 10. The system of claim 1.

[1524] (Claim 3)

[1525] It also includes a means to connect to voice analysis devices used in face-to-face meetings and sales situations.

[1526] 10. The system of claim 1.

[1527] "Example 1"

[1528] (Claim 1)

[1529] means for collecting audio data;

[1530] means for pre-processing to perform noise removal and volume normalization;

[1531] a means for converting the collected voice data into text data using a generative AI model;

[1532] A means for summarizing the generated text data and extracting important information;

[1533] means for storing the summarized data in remote storage;

[1534] a means for displaying real-time feedback;

[1535] A means for analyzing and processing voice data;

[1536] A system including:

[1537] (Claim 2)

[1538] and means for dividing the preprocessed audio data.

[1539] 10. The system of claim 1.

[1540] (Claim 3)

[1541] It also includes a means of linking with voice analysis devices used in face-to-face activities and negotiation situations.

[1542] 10. The system of claim 1.

[1543] "Application Example 1"

[1544] (Claim 1)

[1545] means for collecting audio data;

[1546] means for pre-processing to perform noise removal and volume normalization;

[1547] a means for converting the collected voice data into text data using a generative AI model;

[1548] A means for summarizing the generated text data and extracting important information;

[1549] a means for storing the summarized data in cloud storage;

[1550] a means for displaying real-time feedback;

[1551] means for analyzing voice data transmitted from a voice recognition device;

[1552] A means for store employees to receive analysis results in real time,

[1553] A system including:

[1554] (Claim 2)

[1555] 10. The system of claim 1, further comprising means for chunking the preprocessed audio data.

[1556] (Claim 3)

[1557] The system according to claim 1, further comprising means for linking with a voice analysis device used in face-to-face meetings and business situations.

[1558] "Example 2: Combining Emotion Engines"

[1559] (Claim 1)

[1560] means for collecting audio data;

[1561] means for pre-processing the collected audio data to perform noise reduction and volume normalization;

[1562] a means for converting the preprocessed speech data into text data using a generative AI model;

[1563] A means for summarizing the generated text data and extracting important information;

[1564] means for recognizing emotions from the speech data and tagging the generated text data with the recognized emotions;

[1565] a means for storing the summarized data and sentiment information in cloud storage;

[1566] means for allowing users to access the stored data via a web portal;

[1567] a means for displaying real-time feedback;

[1568] A system including:

[1569] (Claim 2)

[1570] and means for chunking the preprocessed audio data.

[1571] 10. The system of claim 1.

[1572] (Claim 3)

[1573] It also includes a means to connect to voice analysis devices used in face-to-face meetings and sales situations.

[1574] 10. The system of claim 1.

[1575] "Application example 2 when combining emotion engines"

[1576] (Claim 1)

[1577] means for collecting audio data;

[1578] means for pre-processing to perform noise removal and volume normalization;

[1579] a means for converting the collected voice data into text data using a generative AI model;

[1580] A means for summarizing the generated text data and extracting important information;

[1581] means for recognizing emotion from audio data using an emotion engine;

[1582] a means for storing the summarized data and sentiment information in cloud storage;

[1583] a means for displaying real-time feedback;

[1584] A way to use smart glasses to display feedback in real time and instantly understand what customers are saying and feeling;

[1585] A system including:

[1586] (Claim 2)

[1587] and means for chunking the preprocessed audio data.

[1588] 10. The system of claim 1.

[1589] (Claim 3)

[1590] It also includes a means to connect to voice analysis devices used in face-to-face meetings and sales situations.

[1591] 10. The system of claim 1. [Explanation of symbols]

[1592] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting audio data; means for pre-processing to perform noise removal and volume normalization; a means for converting the collected voice data into text data using a generative AI model; A means for summarizing the generated text data and extracting important information; a means for storing the summarized data in cloud storage; a means for displaying real-time feedback; A system including:

2. and means for chunking the preprocessed audio data. The system of claim 1 .

3. It also includes a means to connect to voice analysis devices used in face-to-face meetings and sales situations. The system of claim 1 .

Images