System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting opinions input by a user by voice; means for converting the collected voice into text data; means for transmitting the text data to a server; means for analyzing and summarizing the text data received by the server; means for proposing improvement measures based on the data analyzed by the server; and means for summarizing the proposed improvement measures as a report and transmitting the report to a terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional employee satisfaction surveys mainly involve filling out paper or digital questionnaires, making it difficult to fully capture employees' honest opinions. Furthermore, they are unable to collect opinions in a timely manner and respond in real time, making it difficult to propose quick improvement measures to improve employee satisfaction. The present invention aims to solve these problems and provide a system that improves employee satisfaction. [Means for solving the problem]

[0005] The present invention provides a system that includes means for collecting opinions input by voice from users and means for converting the collected voice into text data. It also includes means for transmitting the text data to a server, and means for the server to analyze and summarize the received text data. It also includes means for the server to propose improvements based on the analyzed data, and means for compiling the proposed improvements into a report and sending it to a terminal. It also includes means for extracting specific topics from the summarized data and means for generating a report based on the proposed improvements and notifying the user, thereby providing a system that can collect employee opinions in a timely manner and quickly provide appropriate improvements.

[0006] A "user" is a person who inputs opinions by voice and provides opinions that are collected by the system.

[0007] "Voice" is voice data uttered by the user and is a means for inputting opinions.

[0008] A "terminal" is a device that collects a user's voice, converts it into text data through a voice recognition function, and transmits the data to a server.

[0009] The "server" is a central processing unit that receives and analyzes text data sent from the terminal, and provides summaries and proposes improvements.

[0010] "Text data" is character data generated from speech converted by a speech recognition function.

[0011] "Analysis" refers to the processing of text data received by the server to summarize opinions and identify topics.

[0012] "Summarization" refers to extracting important information from the analyzed text data and summarizing it briefly.

[0013] "Topics" are major themes or topics extracted from the analyzed text data.

[0014] "Improvement measures" are specific measures or policies proposed based on summarized data or topics.

[0015] A "report" is a document that organizes and documents the proposed improvement measures.

[0016] "Notification" is the act of informing users and other interested parties about the generated report and improvement measures. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] Overall system overview

[0039] This invention is a system that collects opinions entered by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. Specifically, users enter their opinions by voice using a terminal, and the terminal collects and converts the voice data into text. The text data is then sent to a server, where it is analyzed and a summary and improvement proposals are generated. Finally, the text data is sent to the terminal as a report and notified to the user or administrator.

[0040] Program processing overview

[0041] User voice input and collection

[0042] The user can input their opinion by voice. For example, they can say, "I feel like there's been a lack of internal communication recently." The device collects this voice in real time and converts it into text data using its voice recognition function.

[0043] Sending and receiving text data

[0044] The device sends the collected text data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and stores it in a database. For example, the text data stored might read, "I feel like there's been a lack of internal communication recently."

[0045] Server analysis and summarization

[0046] The server analyzes the received text data and uses natural language processing (NLP) algorithms to extract important opinions and key phrases. When similar opinions are collected from multiple users, these are summarized to extract major topics. For example, a topic such as "lack of internal communication" may be extracted.

[0047] Proposal for improvement

[0048] Based on the extracted topics, the server proposes improvement measures, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" are proposed.

[0049] Report preparation and notification

[0050] The server compiles the proposed improvements into a report, which is then sent back to the terminal. The terminal displays the received report through a user interface so that users and administrators can view it. Furthermore, the terminal notifies the user when the report is complete through a notification function.

[0051] Specific examples

[0052] For example, if an employee says, "We're behind on our project, so we'd like to have more frequent meetings," the following process will be executed:

[0053] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0054] 2. The device collects this voice and converts it into text that reads, "The project is progressing slowly, so please hold more frequent meetings."

[0055] 3. The device sends the converted text data to the server.

[0056] 4. The server receives the text data and summarizes it as "increase the frequency of meetings to accelerate project progress."

[0057] 5. Based on the summary data, the server suggests improvement measures such as "increase the frequency of meetings from once a week to twice a week."

[0058] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[0059] 7. The terminal notifies the user of the report, and the user can check the report.

[0060] In this way, the present invention provides a system that performs a series of processes from collecting voice opinions to proposing improvement measures and creating a report.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[0064] Step 2:

[0065] The terminal collects the user's voice using a voice input device such as a microphone.

[0066] Step 3:

[0067] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, it generates text such as "I feel that internal communication has been lacking recently."

[0068] Step 4:

[0069] The device sends the generated text data to the server via an HTTP request or WebSocket.

[0070] Step 5:

[0071] The server receives the text data and stores it in a database, managed in an appropriate format such as SQL or NoSQL.

[0072] Step 6:

[0073] The server analyzes the received text data and uses a natural language processing (NLP) engine to extract important opinions and key phrases.

[0074] Step 7:

[0075] The server summarizes the analysis results. If there are multiple opinions, common themes are extracted and summarized briefly.

[0076] Step 8:

[0077] The server proposes improvement measures based on the summarized data, and generates specific measures by referencing past data and machine learning models.

[0078] Step 9:

[0079] The server will compile the proposed improvements into a report, creating a structured document that lists the improvements and includes detailed explanations.

[0080] Step 10:

[0081] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[0082] Step 11:

[0083] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[0084] Step 12:

[0085] The device notifies the user when the report is complete. The notification content is determined based on business logic via the notification API.

[0086] Through this series of steps, we will create a system that can efficiently collect and analyze user opinions and provide appropriate improvement measures in real time.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] Conventional opinion collection systems lack the ability to effectively convert user voice input into text data, analyze it, and summarize it, resulting in problems that do not lead to improved productivity. Furthermore, the process of proposing improvement measures is not automated, which consumes a lot of time and effort from users, and the process of proposing improvement measures lacks consistency. Considering these issues, there is a need for an efficient and consistent opinion collection and improvement proposal system.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for analyzing received text data using a natural language processing algorithm and extracting key phrases and major topics, means for proposing improvement measures using a generative AI model, and means for compiling the proposed improvement measures into a report and sending it to the terminal. This makes it possible to efficiently convert user speech into text data, extract important opinions and topics, and automatically provide consistent improvement measures.

[0092] A "user" is a person who uses the system to provide voice input and opinions and feedback.

[0093] A "terminal" is a device that allows a user to input voice and has the function of collecting voice and converting it into text data.

[0094] The "server" is a central processing unit that receives and analyzes text data sent from the terminal, proposes improvements, and compiles them into a report.

[0095] "Voice input" refers to the act of a user giving voice to a terminal to give opinions or feedback.

[0096] "Text data" is data in the form of a character string converted based on voice input.

[0097] A "natural language processing algorithm" is a programming technique for analyzing text data and extracting key phrases and major topics.

[0098] A "generative AI model" is an artificial intelligence program that generates new information and suggestions based on pre-trained data.

[0099] "Improvement measures" are specific responses or countermeasures to the extracted opinions or topics.

[0100] A "report" is a document summarizing proposed improvements and is provided to users and administrators.

[0101] "User interface" refers to the display screen and operation means that allow a user to check reports and notifications on a terminal.

[0102] "Notifications" are messages or signals that notify the user of report completion or information from the system.

[0103] The present invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system uses the following hardware and software:

[0104] System configuration

[0105] User voice input and collection

[0106] A user inputs their opinion by voice using a device such as a smartphone, PC, or tablet. For example, the user might say, "I feel like there's been a lack of communication within the company recently." The device collects the voice through a built-in or connected microphone and converts the voice into text data in real time using the Google® Cloud Speech-to-Text API.

[0107] Sending and receiving text data

[0108] The device sends the converted text data to the server. This communication uses standard data transmission methods such as the HTTP protocol or WebSocket. The server stores the received text data in a database (for example, MySQL (registered trademark)). As a specific example, the text data "I feel that there has been a lack of internal communication recently" is sent to the server and stored in the database.

[0109] Server analysis and summarization

[0110] The text data received by the server is analyzed using natural language processing (NLP) algorithms, such as NLTK or spaCy. Important opinions and key phrases are extracted, and similar opinions from multiple users are summarized. For example, a topic such as "lack of internal communication" is extracted.

[0111] Proposal for improvement

[0112] Based on the extracted topics, the server uses a generative AI model (such as GPT-4 (registered trademark)) to propose improvement measures. The server uses pre-trained data and past performance data to generate specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[0113] Report preparation and notification

[0114] The server compiles the proposed improvements into a report and sends it to the device. The created report is then sent back to the device, which displays the received report through a user interface (for example, an interface using React.js). In addition, the device notifies the user of the completion of the report through a notification function.

[0115] Specific examples

[0116] For example, if an employee says, "The project is progressing slowly, so I would like to have more frequent meetings," the process will proceed as follows:

[0117] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0118] 2. The device collects this voice and converts it into text, such as "The project is behind schedule, so please hold more frequent meetings." This is done using the Google Cloud Speech-to-Text API.

[0119] 3. The terminal sends the converted text data to the server using the HTTP protocol.

[0120] 4. The server receives the text data and summarizes it as "Increase the frequency of meetings to accelerate project progress." NLTK and spaCy are used for analysis.

[0121] 5. The server uses the summary data to suggest improvements, such as increasing the frequency of meetings from once a week to twice a week. It uses the generative AI model GPT-4.

[0122] 6. The server compiles the proposed improvements into a report and sends it to the device. React.js is used for the front end.

[0123] 7. The terminal notifies the user of the report, and the user can check the report.

[0124] In this way, the present invention provides a system that efficiently executes a series of processes from collecting voice feedback to proposing improvement measures and creating reports, thereby improving productivity and providing consistent improvement measures.

[0125] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0126] Step 1:

[0127] The user inputs their opinion by voice.

[0128] Specifically, the user speaks to their smartphone or PC, saying, "I feel like there's been a lack of communication within the company recently." This voice is then input into the device.

[0129] Step 2:

[0130] The device collects the audio.

[0131] Specifically, the device's microphone captures the audio and stores it as audio data. This stored audio data is used as input in the next step.

[0132] Step 3:

[0133] The device converts the voice data into text data using the Google Cloud Speech-to-Text API.

[0134] Specifically, the voice data is sent to the API, and a speech recognition algorithm is applied, generating text data such as "I feel that internal communication has been lacking recently."

[0135] Step 4:

[0136] The terminal transmits the converted text data to the server.

[0137] Specifically, the device generates an HTTP POST request and sends it to the server's API endpoint, including the text data as a payload. This HTTP request is input, and the text data is sent to the server.

[0138] Step 5:

[0139] The server receives the text data and stores it in a database.

[0140] Specifically, the server receives an HTTP request and extracts the text data from the payload. This text data is input and stored in a database. The stored text data is used as output in the next step.

[0141] Step 6:

[0142] The server analyzes the text data using natural language processing algorithms.

[0143] Specifically, the server uses NLTK and spaCy to extract key phrases and major topics. This analysis takes text data as input and outputs a list of key phrases and topics. For example, the topic "lack of internal communication" might be extracted.

[0144] Step 7:

[0145] The server uses a generative AI model based on the extracted topics to propose improvement measures.

[0146] Specifically, the server inputs topic data into a generative AI model (such as GPT-4), which generates improvement measures based on pre-trained data. This process generates specific improvement measures as output, such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[0147] Step 8:

[0148] The server compiles the proposed improvements in a report format.

[0149] Specifically, the proposed improvements are formatted as a text document and compiled into a report. The report generated in this step is the output.

[0150] Step 9:

[0151] The server sends the report to the terminal.

[0152] Specifically, the report is attached to the HTTP response and sent to the terminal. During this process, the report becomes input and is sent to the terminal.

[0153] Step 10:

[0154] The terminal receives the report and displays it through a user interface.

[0155] Specifically, the report received by the device is displayed on the UI so that the user can view it. The report is displayed using an interface that uses React.js. The report is input and displayed on the user interface.

[0156] Step 11:

[0157] The terminal notifies the user when the report is complete.

[0158] Specifically, the notification function is used to send a pop-up or email notification to the user to let them know that the report is complete. This allows the user to be aware of the existence of the report. The notification is used as input to notify the user.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] In conventional content distribution services, the challenge is to quickly and accurately collect user feedback and propose specific improvement measures to improve the quality of the service based on that feedback. In particular, there is a need to provide an automated process from collecting voice opinions to analyzing them, proposing improvement measures, and creating reports.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data to the server, means for analyzing and summarizing the text data received by the server, means for proposing improvements based on the analyzed data by the server, means for compiling the proposed improvements as a report and transmitting it to the terminal, and means for collecting and analyzing users' voice opinions and proposing improvements to improve the quality of the service in a content distribution service, thereby enabling high-quality content distribution that quickly reflects user opinions.

[0164] "User" refers to a user who uses the content distribution service and inputs their opinion by voice.

[0165] "Voice input" refers to a user verbally voicing opinions or feedback.

[0166] "Text data" refers to digital data converted from voice input into text information.

[0167] "Server" refers to a computer system that communicates over a network and analyzes, summarizes, and creates reports on received data.

[0168] "Analysis" refers to the process of understanding and classifying received text data using techniques such as natural language processing, and extracting important information.

[0169] "Summarization" refers to the process of summarizing the information obtained through analysis in a concise and easy-to-understand manner.

[0170] "Improvement measures" refer to specific solutions for improving the quality of content distribution services based on collected user opinions.

[0171] "Report" means a document that provides a summary and recommendations for improvement to an administrator or content creator.

[0172] "Terminal" refers to hardware devices that process information, such as computers and smartphones used by users and administrators.

[0173] "Content distribution service" refers to a system that provides digital content such as videos and music via a network.

[0174] "Feedback" refers to opinions and impressions that users give regarding the content distribution service.

[0175] The present invention is a system for collecting user feedback in the form of voice in a content distribution service, analyzing and summarizing the feedback, and proposing and notifying improvement measures. The following describes an embodiment of the present invention.

[0176] Overall system overview

[0177] This system consists of a series of processes to collect users' voice opinions, convert them into text data, analyze them, summarize them, and propose improvements. Specifically, the main components are the device used by the user, voice recognition software, a server that manages and analyzes the text data, and a generative AI model that proposes improvements.

[0178] Hardware Configuration

[0179] Device: A user device such as a smartphone, tablet, or computer.

[0180] Microphone: Used for voice input.

[0181] Server: A high-performance computer system that runs in the cloud or on-premise.

[0182] Software Configuration

[0183] Speech Recognition Library: Uses Python's speech_recognition library to convert speech to text.

[0184] HTTP communication library: Use the requests library to send the converted speech to text data to the server.

[0185] Natural Language Processing Library: Utilizes the transformers library to analyze text data and generate summaries and remediation suggestions.

[0186] Generative AI model: Processes prompt sentences to generate improvements using models such as GPT-3 (registered trademark).

[0187] Specific operation example

[0188] 1. Voice input and collection

[0189] Users can provide feedback to the content distribution service by voice, such as, "The subtitles for this video are displayed late, so I'd like them to be fixed."

[0190] 2. Speech-to-text

[0191] The device uses the speech_recognition library to collect the user's voice in real time and convert it into text data. The voice input data is temporarily stored in the device's memory while the conversion process is performed.

[0192] 3. Sending and receiving text data

[0193] The terminal uses the requests library to send the converted text data to the server via the HTTP protocol, where it is stored and prepared for analysis.

[0194] 4. Server Analysis and Summarization

[0195] The server analyzes the received text data using the transformers library to extract important opinions and key phrases, and generates a summary based on the analysis results, concisely summarizing the main points the user wants to make.

[0196] 5. Proposal for improvement

[0197] The server uses a summary of the text data to suggest improvement measures, taking into account pre-trained data and past performance. The server uses a generative AI model such as GPT-3 to generate the improvement measures by inputting the following prompt sentence:

[0198] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[0199] 6. Preparation and notification of reports

[0200] The server compiles the proposed improvements into a report and sends it to the terminal. The terminal displays the received report through a user interface so that users and administrators can easily check it. It also notifies the user when the report is complete through a notification function.

[0201] The present invention makes it possible to quickly reflect user opinions and deliver high-quality content.

[0202] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0203] Step 1:

[0204] User voice input

[0205] A user inputs their opinion to a content distribution service by voice, for example, "The subtitles for this video are displayed late, so I would like them to be fixed." The user's voice is collected as input, and the output is audio data. Specifically, the system collects the voice in real time using the device's microphone.

[0206] Step 2:

[0207] Converting voice to text data

[0208] The device uses the speech_recognition library to convert speech into text data. The collected speech data is used as input. For data processing, the speech recognition algorithm analyzes the speech data and generates corresponding text. The output is text data. Specifically, the speech recognition software converts speech data into text data.

[0209] Step 3:

[0210] Send text data to the server

[0211] The terminal sends the converted text data to the server using the requests library. The text data generated earlier is used as input. Data is sent using the HTTP protocol. The output is text data stored on the server. Specifically, an HTTP request is sent to a specific endpoint on the server.

[0212] Step 4:

[0213] Server-based analysis and summarization of text data

[0214] The server analyzes and summarizes the received text data using the transformers library. The text data stored on the server is used as input. To process the data, a natural language processing algorithm analyzes the text, extracts important key phrases and opinions, and generates a summary. The output is the summarized text. Specifically, the natural language processing model analyzes the text data and extracts important information.

[0215] Step 5:

[0216] Server-suggested improvement measures

[0217] The server proposes improvement measures based on the summarized text data, taking into account pre-learned data and past performance. The summarized text is used as input. As a data calculation, a generative AI model (e.g., GPT-3) generates improvement measures based on the prompt sentence. The output is the proposed improvement measures. Specifically, the generative AI model operates using the following prompt sentence:

[0218] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[0219] Step 6:

[0220] Report preparation and notification

[0221] The server creates a report including the proposed improvements and sends it to the terminal. The user receives a notification and can view the report. The proposed improvements are used as input. Data processing involves formatting the improvements into a report. The output is the report sent to the terminal and the associated notification. Specifically, the report is formatted in JSON or another format and sent to the terminal using the HTTP protocol. The terminal displays the received report through a user interface and notifies the user.

[0222] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0223] Overall system overview

[0224] This invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system incorporates an emotion engine that recognizes the user's emotions from the voice input and reflects them in the analysis and proposed improvements.

[0225] Program processing overview

[0226] User voice input and collection

[0227] The user inputs their opinion by voice. For example, they might say, "I feel like there's been a lack of communication within the company recently." The device collects this voice in real time and converts it into text data using a voice recognition function. It also uses an emotion engine to recognize the user's emotions. For example, emotional data such as "dissatisfaction" or "stress" is obtained.

[0228] Sending and receiving text data and emotion data

[0229] The device sends the collected text data and emotion data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and emotion data and stores them in a database. For example, text data such as "I feel that there has been a lack of communication within the company recently" and emotion data such as "dissatisfied" are stored.

[0230] Server analysis and summarization

[0231] The server analyzes the received text data and emotion data. It uses a natural language processing (NLP) engine to extract important opinions and key phrases from the text data, and incorporates the emotion data into the analysis. When similar opinions are collected from multiple users, they are summarized and the main topics are extracted along with the emotion data. For example, a topic such as "lack of internal communication" and the associated emotion data of "dissatisfaction" are extracted.

[0232] Proposing improvement measures based on emotion data

[0233] The server then proposes improvement measures based on the extracted topics and emotion data, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" may be proposed. The server takes emotion data into account to generate improvement measures that better reflect the user's emotions.

[0234] Report preparation and notification

[0235] The server compiles the proposed improvements into a report. The report is a structured document that includes a list of improvements and detailed explanations, as well as analysis results based on the emotion data. The report is then sent back to the device, where it is displayed via a user interface for users and administrators to view. The device also notifies users of the completion of the report via a notification function.

[0236] Specific examples

[0237] For example, if an employee says, "The project is behind schedule, so I'd like to have more frequent meetings," and expresses "dissatisfaction," the following process will be executed:

[0238] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0239] 2. The device collects this speech and converts it into text, such as, "The project is behind schedule, so please hold more frequent meetings." The emotion engine recognizes the emotion "dissatisfied."

[0240] 3. The device sends the converted text data and emotion data to the server.

[0241] 4. The server receives the text data and emotion data and summarizes it as "increase the frequency of meetings to accelerate project progress." The emotion data "dissatisfied" is also saved.

[0242] 5. The server uses the summary data and emotion data to propose improvement measures, such as "increasing the frequency of meetings from once a week to twice a week." In this case, more detailed measures are proposed, taking into account the emotion of "dissatisfaction."

[0243] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[0244] 7. The terminal notifies the user of the report, and the user can check the report.

[0245] In this way, the present invention realizes a system that carries out a series of processes from collecting voice opinions to proposing improvement measures and creating reports, and by incorporating emotional data, provides more effective measures to improve employee satisfaction.

[0246] The processing flow will be explained below.

[0247] Step 1:

[0248] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[0249] Step 2:

[0250] The device collects the user's voice. The voice input device (such as a microphone) captures the voice data. At this point, it is saved as an audio file.

[0251] Step 3:

[0252] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, the generated text might say, "I feel like there's been a lack of communication within the company recently." The emotion engine then analyzes the voice data and recognizes the user's emotion (for example, "dissatisfied").

[0253] Step 4:

[0254] The device sends the generated text data and emotion data to the server using an HTTP request or the WebSocket communication protocol.

[0255] Step 5:

[0256] The server receives the transmitted text data and emotion data and stores them in a database. For example, the text "There has been a lack of communication within the company recently" and the emotion data "dissatisfied" are stored.

[0257] Step 6:

[0258] The server analyzes the text and sentiment data received from the database, and uses a natural language processing (NLP) engine to extract important opinions and key phrases, which are then analyzed along with the sentiment data.

[0259] Step 7:

[0260] The server summarizes the results of the analysis. If there are multiple opinions, common themes are extracted and a short summary is made, taking into account the emotion data. For example, the topic "lack of internal communication" and the emotion "dissatisfaction" are summarized.

[0261] Step 8:

[0262] The server proposes improvement measures based on the summarized data and emotion data. Specific measures that take the user's emotions into account are generated based on pre-learned data and past performance. For example, it may suggest "holding regular all-employee meetings" or "introducing specialized communication tools."

[0263] Step 9:

[0264] The server compiles the proposed improvement measures in a report format. A report document is created that includes a list of the improvement measures, detailed explanations, and the designation of the departments and business operations that will be affected. The report also includes the results of the analysis based on the emotion data.

[0265] Step 10:

[0266] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[0267] Step 11:

[0268] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[0269] Step 12:

[0270] The device notifies the user that the report is complete via the notification API.

[0271] Through these detailed steps, we have created a system that efficiently collects users' voice opinions, analyzes them, takes into account emotional data, proposes improvement measures, and finally notifies them in the form of a report.

[0272] Example 2

[0273] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] In today's work environment, lack of internal communication and declining employee satisfaction are major issues. There is a need for an effective system to efficiently collect employee complaints and opinions and propose appropriate improvement measures. However, traditional manual methods of collecting opinions and proposing improvement measures are time-consuming and labor-intensive, and have the problem of not accurately reflecting employee sentiment.

[0275] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0276] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data and emotion data to the server, means for analyzing and summarizing the received text data and emotion data, means for proposing improvement measures based on the analyzed text data and emotion data, and means for compiling the proposed improvement measures into a report and transmitting it to the terminal. This makes it possible to quickly and accurately collect and analyze employee opinions and emotions, propose optimal improvement measures, and improve employee satisfaction and work efficiency.

[0277] "User" refers to a subject who inputs opinions into the system by voice.

[0278] "Voice input" refers to the process of collecting information by a user speaking into a microphone.

[0279] "Opinions" refers to opinions such as evaluations, suggestions, and impressions expressed by users regarding the system.

[0280] "Device" refers to a device that collects voice and converts it into text data, such as a smartphone or computer.

[0281] "Speech recognition software" refers to an application for converting voice data into text data.

[0282] "Emotion engine" refers to software or algorithms for analyzing emotions from a user's voice.

[0283] "Text data" refers to textual information that expresses a user's opinion and is converted by voice recognition software.

[0284] "Emotion data" refers to information that represents the user's emotional state as analyzed by the emotion engine.

[0285] "Server" refers to a computer system that receives, analyzes, stores, and processes data as needed.

[0286] "Data transmission means" refers to means for transmitting text data and emotion data from a terminal to a server.

[0287] The "HTTP protocol" refers to a communication method for sending and receiving data on the web.

[0288] "WebSocket" refers to a protocol that enables two-way communication.

[0289] "Database" refers to a system that allows a server to store and manage data in a structured manner.

[0290] A "natural language processing engine" refers to technology for analyzing text data and extracting important key phrases and opinions.

[0291] "Keyphrases" refer to words or phrases that are considered particularly important in opinion or text data.

[0292] "Summarizing" refers to the process of combining multiple opinions or data into a concise form.

[0293] "Topics" refer to major themes or topics extracted from the analyzed data.

[0294] "Improvement measures" refer to specific countermeasures proposed based on the analysis results and emotional data.

[0295] "Report" refers to a document summarizing proposed improvements and analysis results.

[0296] "Notification" refers to the act of informing a user of a report or important information.

[0297] This system collects opinions entered by users through speech, converts them into text data, analyzes and summarizes them, proposes improvement measures based on the emotional data, and compiles the results into a report. This system uses common devices and software.

[0298] 1. User voice input and collection

[0299] Users input their opinions by voice using devices such as smartphones or PCs. For example, a user might say, "I feel like there's been a lack of communication within the company recently." At this time, the device activates its microphone and collects voice data. The collected voice data is temporarily stored in a buffer.

[0300] 2. Audio Data Processing

[0301] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert voice data into text data in real time. The converted text data is temporarily stored on the device. At the same time, an emotion engine (e.g., IBM Watson® Tone Analyzer) analyzes the user's emotions from the voice data and extracts emotional data such as "dissatisfaction" or "stress."

[0302] 3. Sending text data and emotion data

[0303] The device sends text data and emotion data to the server using the HTTP protocol. The data is converted to JSON format and sent to the server along with the user ID and a timestamp.

[0304] 4. Processing of Data by the Server

[0305] The server stores the received text data and sentiment data in a database (e.g., MySQL). The stored data is used as raw material for analysis. The server then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy or NLTK) to extract key phrases and important opinions.

[0306] 5. Proposing improvement measures based on emotion data

[0307] Based on the analysis results, the server uses a pre-trained generative AI model (e.g., GPT-4) to generate specific improvement measures. For example, it may suggest improvement measures such as "holding regular all-employee meetings" or "introducing specialized communication tools." In doing so, it takes into account emotional data to provide improvement measures that better reflect the user's emotions.

[0308] Example prompt: "The user expressed dissatisfaction by stating, 'I feel like our internal communication has been lacking lately.' Please suggest the best way to improve this."

[0309] 6. Preparation and notification of reports

[0310] The server compiles the improvement measures into a report format and structures the analysis results based on the text data, details of the improvement measures, and sentiment data into a document, which can be exported in formats such as PDF or HTML and sent back to the device.

[0311] The terminal displays the received report through a user interface for the user or administrator to view, and also notifies the user that the report has been completed through a notification function.

[0312] This system makes it possible to efficiently and accurately collect employee complaints and opinions and quickly propose improvement measures appropriate to the situation, which is expected to improve employee satisfaction and work efficiency.

[0313] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0314] Step 1:

[0315] The user inputs their opinion by voice. For example, the user might say, "I feel like there's been a lack of internal communication recently." This becomes the input to the system. The device activates the microphone and collects voice data in real time. The collected voice data is temporarily stored in a buffer. The input voice data becomes the output of this step.

[0316] Step 2:

[0317] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data. It also uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state from the voice data. The input here is the voice data collected in step 1, and the output is text data and emotion data. Specifically, the voice data is first input into the software, converted into text data, and emotion analysis is performed in parallel.

[0318] Step 3:

[0319] The device sends text data and emotion data to the server using the HTTP protocol or WebSocket. The input data is the text data and emotion data generated in step 2. These data are converted into JSON format and sent to the server along with the user ID and timestamp. The output is a confirmation of successful transmission. The specific operation includes the process in which the device converts the data into an appropriate format and sends it over the network.

[0320] Step 4:

[0321] The server stores the received text data and emotion data in a database (e.g., MySQL). The input is the data sent in step 3, and after receiving it, it stores it in the database using an SQL statement. The output is a confirmation that the data was successfully saved to the database. Specifically, the server first establishes a database connection and inserts the received data into the appropriate tables.

[0322] Step 5:

[0323] The server analyzes the stored text data and sentiment data using a natural language processing (NLP) engine (e.g., spaCy or NLTK). The input data is the data stored in the database in step 4, and important key phrases and opinions are extracted from it. The output is the analyzed text data, key phrases, and topics. Specifically, the server reads the data from the database, inputs it into the NLP engine, and generates the analysis results.

[0324] Step 6:

[0325] The server uses a generative AI model (e.g., GPT-4) to propose improvement measures based on the analysis results. The input prompt is the analysis result obtained in step 5, and the output is the generated improvement measures. Specifically, the server inputs the prompt into the generative AI model and compiles the obtained output as the optimal improvement measures.

[0326] Step 7:

[0327] The server compiles the proposed improvements into a report and exports it in PDF or HTML format. The input is the improvements obtained in step 6, along with the necessary text data and emotion data. The output is the completed report. Specifically, the server uses a report generation tool to compile and format all the information and export it as a file.

[0328] Step 8:

[0329] The server sends the completed report back to the terminal. The input data is the report created in step 7. This is sent to the terminal as an HTTP response, and the user is notified. The output is a notification that the report was successfully sent and a notification to the user. Specifically, the server sends the report as an HTTP request, and the terminal receives it, displays it on the user interface, and notifies the user.

[0330] (Application example 2)

[0331] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] In current virtual stores, it is difficult to quickly and accurately collect user feedback and propose appropriate improvement measures based on it. Furthermore, there is a lack of systems that can provide higher-quality service by recognizing, analyzing, and reflecting emotions in user voice. Therefore, there is a need for a system that can collect voice feedback, convert it into text, analyze emotional data, and propose effective improvement measures in a single flow.

[0333] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data and emotional data, and means for collecting users' voice opinions in a virtual store and proposing improvement measures for products based on the emotional data. This makes it possible to instantly collect users' voice opinions and, based on them, to propose appropriate improvement measures that reflect the emotional data.

[0334] A "user" is an entity that uses the system to input voice opinions.

[0335] "Voice feedback" refers to feedback, requests, opinions, etc. that a user provides to the system by voice.

[0336] The "collection means" refers to a mechanism for detecting the user's voice opinions and capturing them as data.

[0337] "Text data" refers to digital data that has been converted from voice opinions into text information.

[0338] "Emotion data" refers to data containing emotion information extracted from a user's voice opinion.

[0339] "Server" refers to a computer system that analyzes collected text data and emotion data and proposes improvement measures.

[0340] "Analysis means" refers to the process by which the server processes text data and emotion data to extract important information and emotions.

[0341] "Summarization" refers to the process of extracting key points from the analyzed data and summarizing them concisely.

[0342] "Means for proposing improvements" refers to the process by which the server generates specific improvement proposals based on the analysis data.

[0343] "Report" refers to a written summary of proposed improvements and analysis results.

[0344] "Notification" refers to the process by which a user is notified of report completion or specific actions.

[0345] "Virtual store" refers to a virtual shopping environment that offers products and services online and is accessible to users.

[0346] "Presentation" refers to the process by which the system displays and provides information and suggestions to the user.

[0347] 1. System Configuration

[0348] This invention is a system based on collecting opinions input by voice from users and converting them into text data and emotion data. The system is composed of a voice recognition unit, emotion recognition unit, data transmission unit, server unit, analysis unit, summarization unit, proposal unit, report generation unit, and notification unit.

[0349] 2. Voice input and collection

[0350] Users use devices such as smart glasses or head-mounted displays to input their opinions by voice. The device collects the voice via a microphone and converts it into text data in real time. Voice recognition software such as the Google Speech Recognition API is used for voice recognition. Emotion recognition uses natural language processing using the BERT model (for example, "cl-tohoku / bert-base-japanese") to extract the user's emotions.

[0351] 3. Transmission of text data and emotional data

[0352] The device sends the collected text data and emotion data to a server using standard data transmission methods such as the HTTP protocol or WebSocket.

[0353] 4. Server Analysis and Summarization

[0354] The server receives the text and emotion data and stores them in a database. It then uses a natural language processing (NLP) engine to analyze the text data and extract important opinions and key phrases. It also summarizes similar opinions from multiple users and extracts key topics. The summaries reflect the emotion data.

[0355] 5. Propose improvements and generate reports

[0356] The server then uses the extracted topics and sentiment data to propose improvement measures, taking into account pre-learned data and past performance. For example, in response to the comment "the price is high," specific improvement measures such as "holding a sale" or "offering a special offer" are proposed. In this process, the server takes into account sentiment data to generate improvement measures that better reflect the user's sentiment. Finally, the server compiles the proposed improvement measures into a report and sends it to the device. Users and administrators can view the report through the user interface.

[0357] 6. Specific Examples

[0358] If a user says "This product is too expensive" in a virtual store and the emotion "unhappy" is recognized, the following process is executed:

[0359] 1. Users express their opinions.

[0360] 2. The device collects the voice and converts it into text. It then performs emotion recognition and obtains the emotion data of "dissatisfied."

[0361] 3. The device sends the text data and emotion data to the server.

[0362] 4. The server analyzes the received data and summarizes it as "the price is high." It also saves the emotional data "dissatisfied."

[0363] 5. The server then proposes improvements, such as "holding a sale," based on the data, taking into account the emotional data.

[0364] 6. The server generates a report and sends it to the device.

[0365] 7. The terminal notifies the user that the report is complete, and the user can review the report.

[0366] Prompt Sentence Examples

[0367] If a user in your virtual store says "this item is too expensive" about a product, and the system recognizes this as a dissatisfied sentiment, how would you adjust the price or suggest a special offer?

[0368] The above is a specific embodiment of the invention.

[0369] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0370] Step 1:

[0371] The user inputs their opinion by voice.

[0372] Specific actions: The user speaks their opinion into a microphone attached to smart glasses or a head-mounted display. For example, they say, "This product is too expensive."

[0373] Input: Audio data

[0374] Output: Collecting audio data

[0375] Step 2:

[0376] The device collects voice opinions and converts them into text data.

[0377] Specific operation: The device converts the collected voice data into text data in real time using the Google Speech Recognition API. For example, the voice data is converted into text data such as "This product is too expensive."

[0378] Input: Audio data

[0379] Output: Text data

[0380] Step 3:

[0381] The terminal acquires emotion data from the text data.

[0382] Specific operation: The device inputs the converted text data into an emotion recognition model such as "cl-tohoku / bert-base-japanese" and performs emotion analysis. For example, from the text data "This product is too expensive," it obtains emotion data such as "dissatisfied."

[0383] Input: Text data

[0384] Output: Emotion data

[0385] Step 4:

[0386] The terminal transmits the text data and the emotion data to the server.

[0387] Specific operation: The device sends text data and emotion data to the server using the HTTP protocol. For example, the text data "This product is too expensive" and the emotion data "Dissatisfied" are sent.

[0388] Input: Text data, emotion data

[0389] Output: Send data to the server

[0390] Step 5:

[0391] The server analyzes the received text data and emotion data and stores them in a database.

[0392] Specific operation: The server analyzes the received data and extracts key phrases using a natural language processing (NLP) engine. It also stores the text data and emotion data in a database. For example, it stores the text data "This product is too expensive" and the emotion data "Dissatisfied."

[0393] Input: Text data, emotion data

[0394] Output: Analysis results, saved in database

[0395] Step 6:

[0396] The server analyzes multiple opinions and performs summarization and topic extraction.

[0397] Specific operation: The server analyzes the received text data and summarizes important opinions and key phrases. It also extracts key topics based on the sentiment data. For example, the topic "expensive" and the sentiment "dissatisfied" are extracted.

[0398] Input: Text data, emotion data

[0399] Output: Summary data, topics

[0400] Step 7:

[0401] The server proposes improvement measures based on the summary data and emotion data.

[0402] Specific actions: The server uses the data it has learned in advance and past performance data to generate specific improvement measures from the analysis data. For example, it may suggest things like "run a sale" or "offer a special offer."

[0403] Input: Summary data, emotion data

[0404] Output: Improvements

[0405] Step 8:

[0406] The server compiles improvement measures into a report and sends it to the terminal.

[0407] Specific operation: The server compiles the proposed improvement measures into a report format and sends it to the terminal using the HTTP protocol. For example, an improvement measure such as "conducting a sale" is included in the report and sent to the terminal.

[0408] Input: Remedy

[0409] Output: Send report

[0410] Step 9:

[0411] The terminal receives the report and displays and notifies it through a user interface.

[0412] Specific operation: The device displays the received report on the user interface and notifies the user of the completion of the report through a notification function. For example, a notification saying "New improvement measures have been proposed" is displayed on the user's device.

[0413] Input: report

[0414] Output: report display, notification

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0418] [Second embodiment]

[0419] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0420] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0426] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0427] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0430] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0431] Overall system overview

[0432] This invention is a system that collects opinions entered by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. Specifically, users enter their opinions by voice using a terminal, and the terminal collects and converts the voice data into text. The text data is then sent to a server, where it is analyzed and a summary and improvement proposals are generated. Finally, the text data is sent to the terminal as a report and notified to the user or administrator.

[0433] Program processing overview

[0434] User voice input and collection

[0435] The user can input their opinion by voice. For example, they can say, "I feel like there's been a lack of internal communication recently." The device collects this voice in real time and converts it into text data using its voice recognition function.

[0436] Sending and receiving text data

[0437] The device sends the collected text data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and stores it in a database. For example, the text data stored might read, "I feel like there's been a lack of internal communication recently."

[0438] Server analysis and summarization

[0439] The server analyzes the received text data and uses natural language processing (NLP) algorithms to extract important opinions and key phrases. When similar opinions are collected from multiple users, these are summarized to extract major topics. For example, a topic such as "lack of internal communication" may be extracted.

[0440] Proposal for improvement

[0441] Based on the extracted topics, the server proposes improvement measures, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" are proposed.

[0442] Report preparation and notification

[0443] The server compiles the proposed improvements into a report, which is then sent back to the terminal. The terminal displays the received report through a user interface so that users and administrators can view it. Furthermore, the terminal notifies the user when the report is complete through a notification function.

[0444] Specific examples

[0445] For example, if an employee says, "We're behind on our project, so we'd like to have more frequent meetings," the following process will be executed:

[0446] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0447] 2. The device collects this voice and converts it into text that reads, "The project is progressing slowly, so please hold more frequent meetings."

[0448] 3. The device sends the converted text data to the server.

[0449] 4. The server receives the text data and summarizes it as "increase the frequency of meetings to accelerate project progress."

[0450] 5. Based on the summary data, the server suggests improvement measures such as "increase the frequency of meetings from once a week to twice a week."

[0451] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[0452] 7. The terminal notifies the user of the report, and the user can check the report.

[0453] In this way, the present invention provides a system that performs a series of processes from collecting voice opinions to proposing improvement measures and creating a report.

[0454] The processing flow will be explained below.

[0455] Step 1:

[0456] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[0457] Step 2:

[0458] The terminal collects the user's voice using a voice input device such as a microphone.

[0459] Step 3:

[0460] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, it generates text such as "I feel that internal communication has been lacking recently."

[0461] Step 4:

[0462] The device sends the generated text data to the server via an HTTP request or WebSocket.

[0463] Step 5:

[0464] The server receives the text data and stores it in a database, managed in an appropriate format such as SQL or NoSQL.

[0465] Step 6:

[0466] The server analyzes the received text data and uses a natural language processing (NLP) engine to extract important opinions and key phrases.

[0467] Step 7:

[0468] The server summarizes the analysis results. If there are multiple opinions, common themes are extracted and summarized briefly.

[0469] Step 8:

[0470] The server proposes improvement measures based on the summarized data, and generates specific measures by referencing past data and machine learning models.

[0471] Step 9:

[0472] The server will compile the proposed improvements into a report, creating a structured document that lists the improvements and includes detailed explanations.

[0473] Step 10:

[0474] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[0475] Step 11:

[0476] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[0477] Step 12:

[0478] The device notifies the user when the report is complete. The notification content is determined based on business logic via the notification API.

[0479] Through this series of steps, we will create a system that can efficiently collect and analyze user opinions and provide appropriate improvement measures in real time.

[0480] Example 1

[0481] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0482] Conventional opinion collection systems lack the ability to effectively convert user voice input into text data, analyze it, and summarize it, resulting in problems that do not lead to improved productivity. Furthermore, the process of proposing improvement measures is not automated, which consumes a lot of time and effort from users, and the process of proposing improvement measures lacks consistency. Considering these issues, there is a need for an efficient and consistent opinion collection and improvement proposal system.

[0483] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0484] In this invention, the server includes means for analyzing received text data using a natural language processing algorithm and extracting key phrases and major topics, means for proposing improvement measures using a generative AI model, and means for compiling the proposed improvement measures into a report and sending it to the terminal. This makes it possible to efficiently convert user speech into text data, extract important opinions and topics, and automatically provide consistent improvement measures.

[0485] A "user" is a person who uses the system to provide voice input and opinions and feedback.

[0486] A "terminal" is a device that allows a user to input voice and has the function of collecting voice and converting it into text data.

[0487] The "server" is a central processing unit that receives and analyzes text data sent from the terminal, proposes improvements, and compiles them into a report.

[0488] "Voice input" refers to the act of a user giving voice to a terminal to give opinions or feedback.

[0489] "Text data" is data in the form of a character string converted based on voice input.

[0490] A "natural language processing algorithm" is a programming technique for analyzing text data and extracting key phrases and major topics.

[0491] A "generative AI model" is an artificial intelligence program that generates new information and suggestions based on pre-trained data.

[0492] "Improvement measures" are specific responses or countermeasures to the extracted opinions or topics.

[0493] A "report" is a document summarizing proposed improvements and is provided to users and administrators.

[0494] "User interface" refers to the display screen and operation means that allow a user to check reports and notifications on a terminal.

[0495] "Notifications" are messages or signals that notify the user of report completion or information from the system.

[0496] The present invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system uses the following hardware and software:

[0497] System configuration

[0498] User voice input and collection

[0499] Users input their opinions by voice using a device such as a smartphone, PC, or tablet. For example, a user might say, "I feel like there's been a lack of internal communication lately." The device collects the voice through a built-in or connected microphone and converts it into text data in real time using the Google Cloud Speech-to-Text API.

[0500] Sending and receiving text data

[0501] The device sends the converted text data to the server. This communication uses standard data transmission methods such as the HTTP protocol or WebSocket. The server stores the received text data in a database (for example, MySQL). As a specific example, the text data "I feel that there has been a lack of internal communication recently" is sent to the server and stored in the database.

[0502] Server analysis and summarization

[0503] The text data received by the server is analyzed using natural language processing (NLP) algorithms, such as NLTK or spaCy. Important opinions and key phrases are extracted, and similar opinions from multiple users are summarized. For example, a topic such as "lack of internal communication" is extracted.

[0504] Proposal for improvement

[0505] Based on the extracted topics, the server uses a generative AI model (such as GPT-4) to propose improvement measures. The server uses pre-trained data and past performance data to generate specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[0506] Report preparation and notification

[0507] The server compiles the proposed improvements into a report and sends it to the device. The created report is then sent back to the device, which displays the received report through a user interface (for example, an interface using React.js). In addition, the device notifies the user of the completion of the report through a notification function.

[0508] Specific examples

[0509] For example, if an employee says, "The project is progressing slowly, so I would like to have more frequent meetings," the process will proceed as follows:

[0510] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0511] 2. The device collects this voice and converts it into text, such as "The project is behind schedule, so please hold more frequent meetings." This is done using the Google Cloud Speech-to-Text API.

[0512] 3. The terminal sends the converted text data to the server using the HTTP protocol.

[0513] 4. The server receives the text data and summarizes it as "Increase the frequency of meetings to accelerate project progress." NLTK and spaCy are used for analysis.

[0514] 5. The server uses the summary data to suggest improvements, such as increasing the frequency of meetings from once a week to twice a week. It uses the generative AI model GPT-4.

[0515] 6. The server compiles the proposed improvements into a report and sends it to the device. React.js is used for the front end.

[0516] 7. The terminal notifies the user of the report, and the user can check the report.

[0517] In this way, the present invention provides a system that efficiently executes a series of processes from collecting voice feedback to proposing improvement measures and creating reports, thereby improving productivity and providing consistent improvement measures.

[0518] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0519] Step 1:

[0520] The user inputs their opinion by voice.

[0521] Specifically, the user speaks to their smartphone or PC, saying, "I feel like there's been a lack of communication within the company recently." This voice is then input into the device.

[0522] Step 2:

[0523] The device collects the audio.

[0524] Specifically, the device's microphone captures the audio and stores it as audio data. This stored audio data is used as input in the next step.

[0525] Step 3:

[0526] The device converts the voice data into text data using the Google Cloud Speech-to-Text API.

[0527] Specifically, the voice data is sent to the API, and a speech recognition algorithm is applied, generating text data such as "I feel that internal communication has been lacking recently."

[0528] Step 4:

[0529] The terminal transmits the converted text data to the server.

[0530] Specifically, the device generates an HTTP POST request and sends it to the server's API endpoint, including the text data as a payload. This HTTP request is input, and the text data is sent to the server.

[0531] Step 5:

[0532] The server receives the text data and stores it in a database.

[0533] Specifically, the server receives an HTTP request and extracts the text data from the payload. This text data is input and stored in a database. The stored text data is used as output in the next step.

[0534] Step 6:

[0535] The server analyzes the text data using natural language processing algorithms.

[0536] Specifically, the server uses NLTK and spaCy to extract key phrases and major topics. This analysis takes text data as input and outputs a list of key phrases and topics. For example, the topic "lack of internal communication" might be extracted.

[0537] Step 7:

[0538] The server uses a generative AI model based on the extracted topics to propose improvement measures.

[0539] Specifically, the server inputs topic data into a generative AI model (such as GPT-4), which generates improvement measures based on pre-trained data. This process generates specific improvement measures as output, such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[0540] Step 8:

[0541] The server compiles the proposed improvements in a report format.

[0542] Specifically, the proposed improvements are formatted as a text document and compiled into a report. The report generated in this step is the output.

[0543] Step 9:

[0544] The server sends the report to the terminal.

[0545] Specifically, the report is attached to the HTTP response and sent to the terminal. During this process, the report becomes input and is sent to the terminal.

[0546] Step 10:

[0547] The terminal receives the report and displays it through a user interface.

[0548] Specifically, the report received by the device is displayed on the UI so that the user can view it. The report is displayed using an interface that uses React.js. The report is input and displayed on the user interface.

[0549] Step 11:

[0550] The terminal notifies the user when the report is complete.

[0551] Specifically, the notification function is used to send a pop-up or email notification to the user to let them know that the report is complete. This allows the user to be aware of the existence of the report. The notification is used as input to notify the user.

[0552] (Application example 1)

[0553] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0554] In conventional content distribution services, the challenge is to quickly and accurately collect user feedback and propose specific improvement measures to improve the quality of the service based on that feedback. In particular, there is a need to provide an automated process from collecting voice opinions to analyzing them, proposing improvement measures, and creating reports.

[0555] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0556] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data to the server, means for analyzing and summarizing the text data received by the server, means for proposing improvements based on the analyzed data by the server, means for compiling the proposed improvements as a report and transmitting it to the terminal, and means for collecting and analyzing users' voice opinions and proposing improvements to improve the quality of the service in a content distribution service, thereby enabling high-quality content distribution that quickly reflects user opinions.

[0557] "User" refers to a user who uses the content distribution service and inputs their opinion by voice.

[0558] "Voice input" refers to a user verbally voicing opinions or feedback.

[0559] "Text data" refers to digital data converted from voice input into text information.

[0560] "Server" refers to a computer system that communicates over a network and analyzes, summarizes, and creates reports on received data.

[0561] "Analysis" refers to the process of understanding and classifying received text data using techniques such as natural language processing, and extracting important information.

[0562] "Summarization" refers to the process of summarizing the information obtained through analysis in a concise and easy-to-understand manner.

[0563] "Improvement measures" refer to specific solutions for improving the quality of content distribution services based on collected user opinions.

[0564] "Report" means a document that provides a summary and recommendations for improvement to an administrator or content creator.

[0565] "Terminal" refers to hardware devices that process information, such as computers and smartphones used by users and administrators.

[0566] "Content distribution service" refers to a system that provides digital content such as videos and music via a network.

[0567] "Feedback" refers to opinions and impressions that users give regarding the content distribution service.

[0568] The present invention is a system for collecting user feedback in the form of voice in a content distribution service, analyzing and summarizing the feedback, and proposing and notifying improvement measures. The following describes an embodiment of the present invention.

[0569] Overall system overview

[0570] This system consists of a series of processes to collect users' voice opinions, convert them into text data, analyze them, summarize them, and propose improvements. Specifically, the main components are the device used by the user, voice recognition software, a server that manages and analyzes the text data, and a generative AI model that proposes improvements.

[0571] Hardware Configuration

[0572] Device: A user device such as a smartphone, tablet, or computer.

[0573] Microphone: Used for voice input.

[0574] Server: A high-performance computer system that runs in the cloud or on-premise.

[0575] Software Configuration

[0576] Speech Recognition Library: Uses Python's speech_recognition library to convert speech to text.

[0577] HTTP communication library: Use the requests library to send the converted speech to text data to the server.

[0578] Natural Language Processing Library: Utilizes the transformers library to analyze text data and generate summaries and remediation suggestions.

[0579] Generative AI model: Processes prompt sentences to generate improvements using models such as GPT-3.

[0580] Specific operation example

[0581] 1. Voice input and collection

[0582] Users can provide feedback to the content distribution service by voice, such as, "The subtitles for this video are displayed late, so I'd like them to be fixed."

[0583] 2. Speech-to-text

[0584] The device uses the speech_recognition library to collect the user's voice in real time and convert it into text data. The voice input data is temporarily stored in the device's memory while the conversion process is performed.

[0585] 3. Sending and receiving text data

[0586] The terminal uses the requests library to send the converted text data to the server via the HTTP protocol, where it is stored and prepared for analysis.

[0587] 4. Server Analysis and Summarization

[0588] The server analyzes the received text data using the transformers library to extract important opinions and key phrases, and generates a summary based on the analysis results, concisely summarizing the main points the user wants to make.

[0589] 5. Proposal for improvement

[0590] The server uses a summary of the text data to suggest improvement measures, taking into account pre-trained data and past performance. The server uses a generative AI model such as GPT-3 to generate the improvement measures by inputting the following prompt sentence:

[0591] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[0592] 6. Preparation and notification of reports

[0593] The server compiles the proposed improvements into a report and sends it to the terminal. The terminal displays the received report through a user interface so that users and administrators can easily check it. It also notifies the user when the report is complete through a notification function.

[0594] The present invention makes it possible to quickly reflect user opinions and deliver high-quality content.

[0595] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0596] Step 1:

[0597] User voice input

[0598] A user inputs their opinion to a content distribution service by voice, for example, "The subtitles for this video are displayed late, so I would like them to be fixed." The user's voice is collected as input, and the output is audio data. Specifically, the system collects the voice in real time using the device's microphone.

[0599] Step 2:

[0600] Converting voice to text data

[0601] The device uses the speech_recognition library to convert speech into text data. The collected speech data is used as input. For data processing, the speech recognition algorithm analyzes the speech data and generates corresponding text. The output is text data. Specifically, the speech recognition software converts speech data into text data.

[0602] Step 3:

[0603] Send text data to the server

[0604] The terminal sends the converted text data to the server using the requests library. The text data generated earlier is used as input. Data is sent using the HTTP protocol. The output is text data stored on the server. Specifically, an HTTP request is sent to a specific endpoint on the server.

[0605] Step 4:

[0606] Server-based analysis and summarization of text data

[0607] The server analyzes and summarizes the received text data using the transformers library. The text data stored on the server is used as input. To process the data, a natural language processing algorithm analyzes the text, extracts important key phrases and opinions, and generates a summary. The output is the summarized text. Specifically, the natural language processing model analyzes the text data and extracts important information.

[0608] Step 5:

[0609] Server-suggested improvement measures

[0610] The server proposes improvement measures based on the summarized text data, taking into account pre-learned data and past performance. The summarized text is used as input. As a data calculation, a generative AI model (e.g., GPT-3) generates improvement measures based on the prompt sentence. The output is the proposed improvement measures. Specifically, the generative AI model operates using the following prompt sentence:

[0611] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[0612] Step 6:

[0613] Report preparation and notification

[0614] The server creates a report including the proposed improvements and sends it to the terminal. The user receives a notification and can view the report. The proposed improvements are used as input. Data processing involves formatting the improvements into a report. The output is the report sent to the terminal and the associated notification. Specifically, the report is formatted in JSON or another format and sent to the terminal using the HTTP protocol. The terminal displays the received report through a user interface and notifies the user.

[0615] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0616] Overall system overview

[0617] This invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system incorporates an emotion engine that recognizes the user's emotions from the voice input and reflects them in the analysis and proposed improvements.

[0618] Program processing overview

[0619] User voice input and collection

[0620] The user inputs their opinion by voice. For example, they might say, "I feel like there's been a lack of communication within the company recently." The device collects this voice in real time and converts it into text data using a voice recognition function. It also uses an emotion engine to recognize the user's emotions. For example, emotional data such as "dissatisfaction" or "stress" is obtained.

[0621] Sending and receiving text data and emotion data

[0622] The device sends the collected text data and emotion data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and emotion data and stores them in a database. For example, text data such as "I feel that there has been a lack of communication within the company recently" and emotion data such as "dissatisfied" are stored.

[0623] Server analysis and summarization

[0624] The server analyzes the received text data and emotion data. It uses a natural language processing (NLP) engine to extract important opinions and key phrases from the text data, and incorporates the emotion data into the analysis. When similar opinions are collected from multiple users, they are summarized and the main topics are extracted along with the emotion data. For example, a topic such as "lack of internal communication" and the associated emotion data of "dissatisfaction" are extracted.

[0625] Proposing improvement measures based on emotion data

[0626] The server then proposes improvement measures based on the extracted topics and emotion data, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" may be proposed. The server takes emotion data into account to generate improvement measures that better reflect the user's emotions.

[0627] Report preparation and notification

[0628] The server compiles the proposed improvements into a report. The report is a structured document that includes a list of improvements and detailed explanations, as well as analysis results based on the emotion data. The report is then sent back to the device, where it is displayed via a user interface for users and administrators to view. The device also notifies users of the completion of the report via a notification function.

[0629] Specific examples

[0630] For example, if an employee says, "The project is behind schedule, so I'd like to have more frequent meetings," and expresses "dissatisfaction," the following process will be executed:

[0631] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0632] 2. The device collects this speech and converts it into text, such as, "The project is behind schedule, so please hold more frequent meetings." The emotion engine recognizes the emotion "dissatisfied."

[0633] 3. The device sends the converted text data and emotion data to the server.

[0634] 4. The server receives the text data and emotion data and summarizes it as "increase the frequency of meetings to accelerate project progress." The emotion data "dissatisfied" is also saved.

[0635] 5. The server uses the summary data and emotion data to propose improvement measures, such as "increasing the frequency of meetings from once a week to twice a week." In this case, more detailed measures are proposed, taking into account the emotion of "dissatisfaction."

[0636] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[0637] 7. The terminal notifies the user of the report, and the user can check the report.

[0638] In this way, the present invention realizes a system that carries out a series of processes from collecting voice opinions to proposing improvement measures and creating reports, and by incorporating emotional data, provides more effective measures to improve employee satisfaction.

[0639] The processing flow will be explained below.

[0640] Step 1:

[0641] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[0642] Step 2:

[0643] The device collects the user's voice. The voice input device (such as a microphone) captures the voice data. At this point, it is saved as an audio file.

[0644] Step 3:

[0645] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, the generated text might say, "I feel like there's been a lack of communication within the company recently." The emotion engine then analyzes the voice data and recognizes the user's emotion (for example, "dissatisfied").

[0646] Step 4:

[0647] The device sends the generated text data and emotion data to the server using an HTTP request or the WebSocket communication protocol.

[0648] Step 5:

[0649] The server receives the transmitted text data and emotion data and stores them in a database. For example, the text "There has been a lack of communication within the company recently" and the emotion data "dissatisfied" are stored.

[0650] Step 6:

[0651] The server analyzes the text and sentiment data received from the database, and uses a natural language processing (NLP) engine to extract important opinions and key phrases, which are then analyzed along with the sentiment data.

[0652] Step 7:

[0653] The server summarizes the results of the analysis. If there are multiple opinions, common themes are extracted and a short summary is made, taking into account the emotion data. For example, the topic "lack of internal communication" and the emotion "dissatisfaction" are summarized.

[0654] Step 8:

[0655] The server proposes improvement measures based on the summarized data and emotion data. Specific measures that take the user's emotions into account are generated based on pre-learned data and past performance. For example, it may suggest "holding regular all-employee meetings" or "introducing specialized communication tools."

[0656] Step 9:

[0657] The server compiles the proposed improvement measures in a report format. A report document is created that includes a list of the improvement measures, detailed explanations, and the designation of the departments and business operations that will be affected. The report also includes the results of the analysis based on the emotion data.

[0658] Step 10:

[0659] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[0660] Step 11:

[0661] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[0662] Step 12:

[0663] The device notifies the user that the report is complete via the notification API.

[0664] Through these detailed steps, we have created a system that efficiently collects users' voice opinions, analyzes them, takes into account emotional data, proposes improvement measures, and finally notifies them in the form of a report.

[0665] Example 2

[0666] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0667] In today's work environment, lack of internal communication and declining employee satisfaction are major issues. There is a need for an effective system to efficiently collect employee complaints and opinions and propose appropriate improvement measures. However, traditional manual methods of collecting opinions and proposing improvement measures are time-consuming and labor-intensive, and have the problem of not accurately reflecting employee sentiment.

[0668] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0669] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data and emotion data to the server, means for analyzing and summarizing the received text data and emotion data, means for proposing improvement measures based on the analyzed text data and emotion data, and means for compiling the proposed improvement measures into a report and transmitting it to the terminal. This makes it possible to quickly and accurately collect and analyze employee opinions and emotions, propose optimal improvement measures, and improve employee satisfaction and work efficiency.

[0670] "User" refers to a subject who inputs opinions into the system by voice.

[0671] "Voice input" refers to the process of collecting information by a user speaking into a microphone.

[0672] "Opinions" refers to opinions such as evaluations, suggestions, and impressions expressed by users regarding the system.

[0673] "Device" refers to a device that collects voice and converts it into text data, such as a smartphone or computer.

[0674] "Speech recognition software" refers to an application for converting voice data into text data.

[0675] "Emotion engine" refers to software or algorithms for analyzing emotions from a user's voice.

[0676] "Text data" refers to textual information that expresses a user's opinion and is converted by voice recognition software.

[0677] "Emotion data" refers to information that represents the user's emotional state as analyzed by the emotion engine.

[0678] "Server" refers to a computer system that receives, analyzes, stores, and processes data as needed.

[0679] "Data transmission means" refers to means for transmitting text data and emotion data from a terminal to a server.

[0680] The "HTTP protocol" refers to a communication method for sending and receiving data on the web.

[0681] "WebSocket" refers to a protocol that enables two-way communication.

[0682] "Database" refers to a system that allows a server to store and manage data in a structured manner.

[0683] A "natural language processing engine" refers to technology for analyzing text data and extracting important key phrases and opinions.

[0684] "Keyphrases" refer to words or phrases that are considered particularly important in opinion or text data.

[0685] "Summarizing" refers to the process of combining multiple opinions or data into a concise form.

[0686] "Topics" refer to major themes or topics extracted from the analyzed data.

[0687] "Improvement measures" refer to specific countermeasures proposed based on the analysis results and emotional data.

[0688] "Report" refers to a document summarizing proposed improvements and analysis results.

[0689] "Notification" refers to the act of informing a user of a report or important information.

[0690] This system collects opinions entered by users through speech, converts them into text data, analyzes and summarizes them, proposes improvement measures based on the emotional data, and compiles the results into a report. This system uses common devices and software.

[0691] 1. User voice input and collection

[0692] Users input their opinions by voice using devices such as smartphones or PCs. For example, a user might say, "I feel like there's been a lack of communication within the company recently." At this time, the device activates its microphone and collects voice data. The collected voice data is temporarily stored in a buffer.

[0693] 2. Audio Data Processing

[0694] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert voice data into text data in real time. The converted text data is temporarily stored on the device. At the same time, an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the user's emotions from the voice data and extracts emotional data such as "dissatisfaction" or "stress."

[0695] 3. Sending text data and emotion data

[0696] The device sends text data and emotion data to the server using the HTTP protocol. The data is converted to JSON format and sent to the server along with the user ID and a timestamp.

[0697] 4. Processing of Data by the Server

[0698] The server stores the received text data and sentiment data in a database (e.g., MySQL). The stored data is used as raw material for analysis. The server then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy or NLTK) to extract key phrases and important opinions.

[0699] 5. Proposing improvement measures based on emotion data

[0700] Based on the analysis results, the server uses a pre-trained generative AI model (e.g., GPT-4) to generate specific improvement measures. For example, it may suggest improvement measures such as "holding regular all-employee meetings" or "introducing specialized communication tools." In doing so, it takes into account emotional data to provide improvement measures that better reflect the user's emotions.

[0701] Example prompt: "The user expressed dissatisfaction by stating, 'I feel like our internal communication has been lacking lately.' Please suggest the best way to improve this."

[0702] 6. Preparation and notification of reports

[0703] The server compiles the improvement measures into a report format and structures the analysis results based on the text data, details of the improvement measures, and sentiment data into a document, which can be exported in formats such as PDF or HTML and sent back to the device.

[0704] The terminal displays the received report through a user interface for the user or administrator to view, and also notifies the user that the report has been completed through a notification function.

[0705] This system makes it possible to efficiently and accurately collect employee complaints and opinions and quickly propose improvement measures appropriate to the situation, which is expected to improve employee satisfaction and work efficiency.

[0706] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0707] Step 1:

[0708] The user inputs their opinion by voice. For example, the user might say, "I feel like there's been a lack of internal communication recently." This becomes the input to the system. The device activates the microphone and collects voice data in real time. The collected voice data is temporarily stored in a buffer. The input voice data becomes the output of this step.

[0709] Step 2:

[0710] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data. It also uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state from the voice data. The input here is the voice data collected in step 1, and the output is text data and emotion data. Specifically, the voice data is first input into the software, converted into text data, and emotion analysis is performed in parallel.

[0711] Step 3:

[0712] The device sends text data and emotion data to the server using the HTTP protocol or WebSocket. The input data is the text data and emotion data generated in step 2. These data are converted into JSON format and sent to the server along with the user ID and timestamp. The output is a confirmation of successful transmission. The specific operation includes the process in which the device converts the data into an appropriate format and sends it over the network.

[0713] Step 4:

[0714] The server stores the received text data and emotion data in a database (e.g., MySQL). The input is the data sent in step 3, and after receiving it, it stores it in the database using an SQL statement. The output is a confirmation that the data was successfully saved to the database. Specifically, the server first establishes a database connection and inserts the received data into the appropriate tables.

[0715] Step 5:

[0716] The server analyzes the stored text data and sentiment data using a natural language processing (NLP) engine (e.g., spaCy or NLTK). The input data is the data stored in the database in step 4, and important key phrases and opinions are extracted from it. The output is the analyzed text data, key phrases, and topics. Specifically, the server reads the data from the database, inputs it into the NLP engine, and generates the analysis results.

[0717] Step 6:

[0718] The server uses a generative AI model (e.g., GPT-4) to propose improvement measures based on the analysis results. The input prompt is the analysis result obtained in step 5, and the output is the generated improvement measures. Specifically, the server inputs the prompt into the generative AI model and compiles the obtained output as the optimal improvement measures.

[0719] Step 7:

[0720] The server compiles the proposed improvements into a report and exports it in PDF or HTML format. The input is the improvements obtained in step 6, along with the necessary text data and emotion data. The output is the completed report. Specifically, the server uses a report generation tool to compile and format all the information and export it as a file.

[0721] Step 8:

[0722] The server sends the completed report back to the terminal. The input data is the report created in step 7. This is sent to the terminal as an HTTP response, and the user is notified. The output is a notification that the report was successfully sent and a notification to the user. Specifically, the server sends the report as an HTTP request, and the terminal receives it, displays it on the user interface, and notifies the user.

[0723] (Application example 2)

[0724] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0725] In current virtual stores, it is difficult to quickly and accurately collect user feedback and propose appropriate improvement measures based on it. Furthermore, there is a lack of systems that can provide higher-quality service by recognizing, analyzing, and reflecting emotions in user voice. Therefore, there is a need for a system that can collect voice feedback, convert it into text, analyze emotional data, and propose effective improvement measures in a single flow.

[0726] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data and emotional data, and means for collecting users' voice opinions in a virtual store and proposing improvement measures for products based on the emotional data. This makes it possible to instantly collect users' voice opinions and, based on them, to propose appropriate improvement measures that reflect the emotional data.

[0727] A "user" is an entity that uses the system to input voice opinions.

[0728] "Voice feedback" refers to feedback, requests, opinions, etc. that a user provides to the system by voice.

[0729] The "collection means" refers to a mechanism for detecting the user's voice opinions and capturing them as data.

[0730] "Text data" refers to digital data that has been converted from voice opinions into text information.

[0731] "Emotion data" refers to data containing emotion information extracted from a user's voice opinion.

[0732] "Server" refers to a computer system that analyzes collected text data and emotion data and proposes improvement measures.

[0733] "Analysis means" refers to the process by which the server processes text data and emotion data to extract important information and emotions.

[0734] "Summarization" refers to the process of extracting key points from the analyzed data and summarizing them concisely.

[0735] "Means for proposing improvements" refers to the process by which the server generates specific improvement proposals based on the analysis data.

[0736] "Report" refers to a written summary of proposed improvements and analysis results.

[0737] "Notification" refers to the process by which a user is notified of report completion or specific actions.

[0738] "Virtual store" refers to a virtual shopping environment that offers products and services online and is accessible to users.

[0739] "Presentation" refers to the process by which the system displays and provides information and suggestions to the user.

[0740] 1. System Configuration

[0741] This invention is a system based on collecting opinions input by voice from users and converting them into text data and emotion data. The system is composed of a voice recognition unit, emotion recognition unit, data transmission unit, server unit, analysis unit, summarization unit, proposal unit, report generation unit, and notification unit.

[0742] 2. Voice input and collection

[0743] Users use devices such as smart glasses or head-mounted displays to input their opinions by voice. The device collects the voice via a microphone and converts it into text data in real time. Voice recognition software such as the Google Speech Recognition API is used for voice recognition. Emotion recognition uses natural language processing using the BERT model (for example, "cl-tohoku / bert-base-japanese") to extract the user's emotions.

[0744] 3. Transmission of text data and emotional data

[0745] The device sends the collected text data and emotion data to a server using standard data transmission methods such as the HTTP protocol or WebSocket.

[0746] 4. Server Analysis and Summarization

[0747] The server receives the text and emotion data and stores them in a database. It then uses a natural language processing (NLP) engine to analyze the text data and extract important opinions and key phrases. It also summarizes similar opinions from multiple users and extracts key topics. The summaries reflect the emotion data.

[0748] 5. Propose improvements and generate reports

[0749] The server then uses the extracted topics and sentiment data to propose improvement measures, taking into account pre-learned data and past performance. For example, in response to the comment "the price is high," specific improvement measures such as "holding a sale" or "offering a special offer" are proposed. In this process, the server takes into account sentiment data to generate improvement measures that better reflect the user's sentiment. Finally, the server compiles the proposed improvement measures into a report and sends it to the device. Users and administrators can view the report through the user interface.

[0750] 6. Specific Examples

[0751] If a user says "This product is too expensive" in a virtual store and the emotion "unhappy" is recognized, the following process is executed:

[0752] 1. Users express their opinions.

[0753] 2. The device collects the voice and converts it into text. It then performs emotion recognition and obtains the emotion data of "dissatisfied."

[0754] 3. The device sends the text data and emotion data to the server.

[0755] 4. The server analyzes the received data and summarizes it as "the price is high." It also saves the emotional data "dissatisfied."

[0756] 5. The server then proposes improvements, such as "holding a sale," based on the data, taking into account the emotional data.

[0757] 6. The server generates a report and sends it to the device.

[0758] 7. The terminal notifies the user that the report is complete, and the user can review the report.

[0759] Prompt Sentence Examples

[0760] If a user in your virtual store says "this item is too expensive" about a product, and the system recognizes this as a dissatisfied sentiment, how would you adjust the price or suggest a special offer?

[0761] The above is a specific embodiment of the invention.

[0762] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0763] Step 1:

[0764] The user inputs their opinion by voice.

[0765] Specific actions: The user speaks their opinion into a microphone attached to smart glasses or a head-mounted display. For example, they say, "This product is too expensive."

[0766] Input: Audio data

[0767] Output: Collecting audio data

[0768] Step 2:

[0769] The device collects voice opinions and converts them into text data.

[0770] Specific operation: The device converts the collected voice data into text data in real time using the Google Speech Recognition API. For example, the voice data is converted into text data such as "This product is too expensive."

[0771] Input: Audio data

[0772] Output: Text data

[0773] Step 3:

[0774] The terminal acquires emotion data from the text data.

[0775] Specific operation: The device inputs the converted text data into an emotion recognition model such as "cl-tohoku / bert-base-japanese" and performs emotion analysis. For example, from the text data "This product is too expensive," it obtains emotion data such as "dissatisfied."

[0776] Input: Text data

[0777] Output: Emotion data

[0778] Step 4:

[0779] The terminal transmits the text data and the emotion data to the server.

[0780] Specific operation: The device sends text data and emotion data to the server using the HTTP protocol. For example, the text data "This product is too expensive" and the emotion data "Dissatisfied" are sent.

[0781] Input: Text data, emotion data

[0782] Output: Send data to the server

[0783] Step 5:

[0784] The server analyzes the received text data and emotion data and stores them in a database.

[0785] Specific operation: The server analyzes the received data and extracts key phrases using a natural language processing (NLP) engine. It also stores the text data and emotion data in a database. For example, it stores the text data "This product is too expensive" and the emotion data "Dissatisfied."

[0786] Input: Text data, emotion data

[0787] Output: Analysis results, saved in database

[0788] Step 6:

[0789] The server analyzes multiple opinions and performs summarization and topic extraction.

[0790] Specific operation: The server analyzes the received text data and summarizes important opinions and key phrases. It also extracts key topics based on the sentiment data. For example, the topic "expensive" and the sentiment "dissatisfied" are extracted.

[0791] Input: Text data, emotion data

[0792] Output: Summary data, topics

[0793] Step 7:

[0794] The server proposes improvement measures based on the summary data and emotion data.

[0795] Specific actions: The server uses the data it has learned in advance and past performance data to generate specific improvement measures from the analysis data. For example, it may suggest things like "run a sale" or "offer a special offer."

[0796] Input: Summary data, emotion data

[0797] Output: Improvements

[0798] Step 8:

[0799] The server compiles improvement measures into a report and sends it to the terminal.

[0800] Specific operation: The server compiles the proposed improvement measures into a report format and sends it to the terminal using the HTTP protocol. For example, an improvement measure such as "conducting a sale" is included in the report and sent to the terminal.

[0801] Input: Remedy

[0802] Output: Send report

[0803] Step 9:

[0804] The terminal receives the report and displays and notifies it through a user interface.

[0805] Specific operation: The device displays the received report on the user interface and notifies the user of the completion of the report through a notification function. For example, a notification saying "New improvement measures have been proposed" is displayed on the user's device.

[0806] Input: report

[0807] Output: report display, notification

[0808] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0809] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0810] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0811] [Third embodiment]

[0812] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0813] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0814] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0815] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0816] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0817] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0818] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0819] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0820] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0821] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0822] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0823] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0824] Overall system overview

[0825] This invention is a system that collects opinions entered by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. Specifically, users enter their opinions by voice using a terminal, and the terminal collects and converts the voice data into text. The text data is then sent to a server, where it is analyzed and a summary and improvement proposals are generated. Finally, the text data is sent to the terminal as a report and notified to the user or administrator.

[0826] Program processing overview

[0827] User voice input and collection

[0828] The user can input their opinion by voice. For example, they can say, "I feel like there's been a lack of internal communication recently." The device collects this voice in real time and converts it into text data using its voice recognition function.

[0829] Sending and receiving text data

[0830] The device sends the collected text data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and stores it in a database. For example, the text data stored might read, "I feel like there's been a lack of internal communication recently."

[0831] Server analysis and summarization

[0832] The server analyzes the received text data and uses natural language processing (NLP) algorithms to extract important opinions and key phrases. When similar opinions are collected from multiple users, these are summarized to extract major topics. For example, a topic such as "lack of internal communication" may be extracted.

[0833] Proposal for improvement

[0834] Based on the extracted topics, the server proposes improvement measures, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" are proposed.

[0835] Report preparation and notification

[0836] The server compiles the proposed improvements into a report, which is then sent back to the terminal. The terminal displays the received report through a user interface so that users and administrators can view it. Furthermore, the terminal notifies the user when the report is complete through a notification function.

[0837] Specific examples

[0838] For example, if an employee says, "We're behind on our project, so we'd like to have more frequent meetings," the following process will be executed:

[0839] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0840] 2. The device collects this voice and converts it into text that reads, "The project is progressing slowly, so please hold more frequent meetings."

[0841] 3. The device sends the converted text data to the server.

[0842] 4. The server receives the text data and summarizes it as "increase the frequency of meetings to accelerate project progress."

[0843] 5. Based on the summary data, the server suggests improvement measures such as "increase the frequency of meetings from once a week to twice a week."

[0844] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[0845] 7. The terminal notifies the user of the report, and the user can check the report.

[0846] In this way, the present invention provides a system that performs a series of processes from collecting voice opinions to proposing improvement measures and creating a report.

[0847] The processing flow will be explained below.

[0848] Step 1:

[0849] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[0850] Step 2:

[0851] The terminal collects the user's voice using a voice input device such as a microphone.

[0852] Step 3:

[0853] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, it generates text such as "I feel that internal communication has been lacking recently."

[0854] Step 4:

[0855] The device sends the generated text data to the server via an HTTP request or WebSocket.

[0856] Step 5:

[0857] The server receives the text data and stores it in a database, managed in an appropriate format such as SQL or NoSQL.

[0858] Step 6:

[0859] The server analyzes the received text data and uses a natural language processing (NLP) engine to extract important opinions and key phrases.

[0860] Step 7:

[0861] The server summarizes the analysis results. If there are multiple opinions, common themes are extracted and summarized briefly.

[0862] Step 8:

[0863] The server proposes improvement measures based on the summarized data, and generates specific measures by referencing past data and machine learning models.

[0864] Step 9:

[0865] The server will compile the proposed improvements into a report, creating a structured document that lists the improvements and includes detailed explanations.

[0866] Step 10:

[0867] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[0868] Step 11:

[0869] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[0870] Step 12:

[0871] The device notifies the user when the report is complete. The notification content is determined based on business logic via the notification API.

[0872] Through this series of steps, we will create a system that can efficiently collect and analyze user opinions and provide appropriate improvement measures in real time.

[0873] Example 1

[0874] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0875] Conventional opinion collection systems lack the ability to effectively convert user voice input into text data, analyze it, and summarize it, resulting in problems that do not lead to improved productivity. Furthermore, the process of proposing improvement measures is not automated, which consumes a lot of time and effort from users, and the process of proposing improvement measures lacks consistency. Considering these issues, there is a need for an efficient and consistent opinion collection and improvement proposal system.

[0876] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0877] In this invention, the server includes means for analyzing received text data using a natural language processing algorithm and extracting key phrases and major topics, means for proposing improvement measures using a generative AI model, and means for compiling the proposed improvement measures into a report and sending it to the terminal. This makes it possible to efficiently convert user speech into text data, extract important opinions and topics, and automatically provide consistent improvement measures.

[0878] A "user" is a person who uses the system to provide voice input and opinions and feedback.

[0879] A "terminal" is a device that allows a user to input voice and has the function of collecting voice and converting it into text data.

[0880] The "server" is a central processing unit that receives and analyzes text data sent from the terminal, proposes improvements, and compiles them into a report.

[0881] "Voice input" refers to the act of a user giving voice to a terminal to give opinions or feedback.

[0882] "Text data" is data in the form of a character string converted based on voice input.

[0883] A "natural language processing algorithm" is a programming technique for analyzing text data and extracting key phrases and major topics.

[0884] A "generative AI model" is an artificial intelligence program that generates new information and suggestions based on pre-trained data.

[0885] "Improvement measures" are specific responses or countermeasures to the extracted opinions or topics.

[0886] A "report" is a document summarizing proposed improvements and is provided to users and administrators.

[0887] "User interface" refers to the display screen and operation means that allow a user to check reports and notifications on a terminal.

[0888] "Notifications" are messages or signals that notify the user of report completion or information from the system.

[0889] The present invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system uses the following hardware and software:

[0890] System configuration

[0891] User voice input and collection

[0892] Users input their opinions by voice using a device such as a smartphone, PC, or tablet. For example, a user might say, "I feel like there's been a lack of internal communication lately." The device collects the voice through a built-in or connected microphone and converts it into text data in real time using the Google Cloud Speech-to-Text API.

[0893] Sending and receiving text data

[0894] The device sends the converted text data to the server. This communication uses standard data transmission methods such as the HTTP protocol or WebSocket. The server stores the received text data in a database (for example, MySQL). As a specific example, the text data "I feel that there has been a lack of internal communication recently" is sent to the server and stored in the database.

[0895] Server analysis and summarization

[0896] The text data received by the server is analyzed using natural language processing (NLP) algorithms, such as NLTK or spaCy. Important opinions and key phrases are extracted, and similar opinions from multiple users are summarized. For example, a topic such as "lack of internal communication" is extracted.

[0897] Proposal for improvement

[0898] Based on the extracted topics, the server uses a generative AI model (such as GPT-4) to propose improvement measures. The server uses pre-trained data and past performance data to generate specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[0899] Report preparation and notification

[0900] The server compiles the proposed improvements into a report and sends it to the device. The created report is then sent back to the device, which displays the received report through a user interface (for example, an interface using React.js). In addition, the device notifies the user of the completion of the report through a notification function.

[0901] Specific examples

[0902] For example, if an employee says, "The project is progressing slowly, so I would like to have more frequent meetings," the process will proceed as follows:

[0903] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[0904] 2. The device collects this voice and converts it into text, such as "The project is behind schedule, so please hold more frequent meetings." This is done using the Google Cloud Speech-to-Text API.

[0905] 3. The terminal sends the converted text data to the server using the HTTP protocol.

[0906] 4. The server receives the text data and summarizes it as "Increase the frequency of meetings to accelerate project progress." NLTK and spaCy are used for analysis.

[0907] 5. The server uses the summary data to suggest improvements, such as increasing the frequency of meetings from once a week to twice a week. It uses the generative AI model GPT-4.

[0908] 6. The server compiles the proposed improvements into a report and sends it to the device. React.js is used for the front end.

[0909] 7. The terminal notifies the user of the report, and the user can check the report.

[0910] In this way, the present invention provides a system that efficiently executes a series of processes from collecting voice feedback to proposing improvement measures and creating reports, thereby improving productivity and providing consistent improvement measures.

[0911] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0912] Step 1:

[0913] The user inputs their opinion by voice.

[0914] Specifically, the user speaks to their smartphone or PC, saying, "I feel like there's been a lack of communication within the company recently." This voice is then input into the device.

[0915] Step 2:

[0916] The device collects the audio.

[0917] Specifically, the device's microphone captures the audio and stores it as audio data. This stored audio data is used as input in the next step.

[0918] Step 3:

[0919] The device converts the voice data into text data using the Google Cloud Speech-to-Text API.

[0920] Specifically, the voice data is sent to the API, and a speech recognition algorithm is applied, generating text data such as "I feel that internal communication has been lacking recently."

[0921] Step 4:

[0922] The terminal transmits the converted text data to the server.

[0923] Specifically, the device generates an HTTP POST request and sends it to the server's API endpoint, including the text data as a payload. This HTTP request is input, and the text data is sent to the server.

[0924] Step 5:

[0925] The server receives the text data and stores it in a database.

[0926] Specifically, the server receives an HTTP request and extracts the text data from the payload. This text data is input and stored in a database. The stored text data is used as output in the next step.

[0927] Step 6:

[0928] The server analyzes the text data using natural language processing algorithms.

[0929] Specifically, the server uses NLTK and spaCy to extract key phrases and major topics. This analysis takes text data as input and outputs a list of key phrases and topics. For example, the topic "lack of internal communication" might be extracted.

[0930] Step 7:

[0931] The server uses a generative AI model based on the extracted topics to propose improvement measures.

[0932] Specifically, the server inputs topic data into a generative AI model (such as GPT-4), which generates improvement measures based on pre-trained data. This process generates specific improvement measures as output, such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[0933] Step 8:

[0934] The server compiles the proposed improvements in a report format.

[0935] Specifically, the proposed improvements are formatted as a text document and compiled into a report. The report generated in this step is the output.

[0936] Step 9:

[0937] The server sends the report to the terminal.

[0938] Specifically, the report is attached to the HTTP response and sent to the terminal. During this process, the report becomes input and is sent to the terminal.

[0939] Step 10:

[0940] The terminal receives the report and displays it through a user interface.

[0941] Specifically, the report received by the device is displayed on the UI so that the user can view it. The report is displayed using an interface that uses React.js. The report is input and displayed on the user interface.

[0942] Step 11:

[0943] The terminal notifies the user when the report is complete.

[0944] Specifically, the notification function is used to send a pop-up or email notification to the user to let them know that the report is complete. This allows the user to be aware of the existence of the report. The notification is used as input to notify the user.

[0945] (Application example 1)

[0946] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0947] In conventional content distribution services, the challenge is to quickly and accurately collect user feedback and propose specific improvement measures to improve the quality of the service based on that feedback. In particular, there is a need to provide an automated process from collecting voice opinions to analyzing them, proposing improvement measures, and creating reports.

[0948] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0949] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data to the server, means for analyzing and summarizing the text data received by the server, means for proposing improvements based on the analyzed data by the server, means for compiling the proposed improvements as a report and transmitting it to the terminal, and means for collecting and analyzing users' voice opinions and proposing improvements to improve the quality of the service in a content distribution service, thereby enabling high-quality content distribution that quickly reflects user opinions.

[0950] "User" refers to a user who uses the content distribution service and inputs their opinion by voice.

[0951] "Voice input" refers to a user verbally voicing opinions or feedback.

[0952] "Text data" refers to digital data converted from voice input into text information.

[0953] "Server" refers to a computer system that communicates over a network and analyzes, summarizes, and creates reports on received data.

[0954] "Analysis" refers to the process of understanding and classifying received text data using techniques such as natural language processing, and extracting important information.

[0955] "Summarization" refers to the process of summarizing the information obtained through analysis in a concise and easy-to-understand manner.

[0956] "Improvement measures" refer to specific solutions for improving the quality of content distribution services based on collected user opinions.

[0957] "Report" means a document that provides a summary and recommendations for improvement to an administrator or content creator.

[0958] "Terminal" refers to hardware devices that process information, such as computers and smartphones used by users and administrators.

[0959] "Content distribution service" refers to a system that provides digital content such as videos and music via a network.

[0960] "Feedback" refers to opinions and impressions that users give regarding the content distribution service.

[0961] The present invention is a system for collecting user feedback in the form of voice in a content distribution service, analyzing and summarizing the feedback, and proposing and notifying improvement measures. The following describes an embodiment of the present invention.

[0962] Overall system overview

[0963] This system consists of a series of processes to collect users' voice opinions, convert them into text data, analyze them, summarize them, and propose improvements. Specifically, the main components are the device used by the user, voice recognition software, a server that manages and analyzes the text data, and a generative AI model that proposes improvements.

[0964] Hardware Configuration

[0965] Device: A user device such as a smartphone, tablet, or computer.

[0966] Microphone: Used for voice input.

[0967] Server: A high-performance computer system that runs in the cloud or on-premise.

[0968] Software Configuration

[0969] Speech Recognition Library: Uses Python's speech_recognition library to convert speech to text.

[0970] HTTP communication library: Use the requests library to send the converted speech to text data to the server.

[0971] Natural Language Processing Library: Utilizes the transformers library to analyze text data and generate summaries and remediation suggestions.

[0972] Generative AI model: Processes prompt sentences to generate improvements using models such as GPT-3.

[0973] Specific operation example

[0974] 1. Voice input and collection

[0975] Users can provide feedback to the content distribution service by voice, such as, "The subtitles for this video are displayed late, so I'd like them to be fixed."

[0976] 2. Speech-to-text

[0977] The device uses the speech_recognition library to collect the user's voice in real time and convert it into text data. The voice input data is temporarily stored in the device's memory while the conversion process is performed.

[0978] 3. Sending and receiving text data

[0979] The terminal uses the requests library to send the converted text data to the server via the HTTP protocol, where it is stored and prepared for analysis.

[0980] 4. Server Analysis and Summarization

[0981] The server analyzes the received text data using the transformers library to extract important opinions and key phrases, and generates a summary based on the analysis results, concisely summarizing the main points the user wants to make.

[0982] 5. Proposal for improvement

[0983] The server uses a summary of the text data to suggest improvement measures, taking into account pre-trained data and past performance. The server uses a generative AI model such as GPT-3 to generate the improvement measures by inputting the following prompt sentence:

[0984] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[0985] 6. Preparation and notification of reports

[0986] The server compiles the proposed improvements into a report and sends it to the terminal. The terminal displays the received report through a user interface so that users and administrators can easily check it. It also notifies the user when the report is complete through a notification function.

[0987] The present invention makes it possible to quickly reflect user opinions and deliver high-quality content.

[0988] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0989] Step 1:

[0990] User voice input

[0991] A user inputs their opinion to a content distribution service by voice, for example, "The subtitles for this video are displayed late, so I would like them to be fixed." The user's voice is collected as input, and the output is audio data. Specifically, the system collects the voice in real time using the device's microphone.

[0992] Step 2:

[0993] Converting voice to text data

[0994] The device uses the speech_recognition library to convert speech into text data. The collected speech data is used as input. For data processing, the speech recognition algorithm analyzes the speech data and generates corresponding text. The output is text data. Specifically, the speech recognition software converts speech data into text data.

[0995] Step 3:

[0996] Send text data to the server

[0997] The terminal sends the converted text data to the server using the requests library. The text data generated earlier is used as input. Data is sent using the HTTP protocol. The output is text data stored on the server. Specifically, an HTTP request is sent to a specific endpoint on the server.

[0998] Step 4:

[0999] Server-based analysis and summarization of text data

[1000] The server analyzes and summarizes the received text data using the transformers library. The text data stored on the server is used as input. To process the data, a natural language processing algorithm analyzes the text, extracts important key phrases and opinions, and generates a summary. The output is the summarized text. Specifically, the natural language processing model analyzes the text data and extracts important information.

[1001] Step 5:

[1002] Server-suggested improvement measures

[1003] The server proposes improvement measures based on the summarized text data, taking into account pre-learned data and past performance. The summarized text is used as input. As a data calculation, a generative AI model (e.g., GPT-3) generates improvement measures based on the prompt sentence. The output is the proposed improvement measures. Specifically, the generative AI model operates using the following prompt sentence:

[1004] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[1005] Step 6:

[1006] Report preparation and notification

[1007] The server creates a report including the proposed improvements and sends it to the terminal. The user receives a notification and can view the report. The proposed improvements are used as input. Data processing involves formatting the improvements into a report. The output is the report sent to the terminal and the associated notification. Specifically, the report is formatted in JSON or another format and sent to the terminal using the HTTP protocol. The terminal displays the received report through a user interface and notifies the user.

[1008] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1009] Overall system overview

[1010] This invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system incorporates an emotion engine that recognizes the user's emotions from the voice input and reflects them in the analysis and proposed improvements.

[1011] Program processing overview

[1012] User voice input and collection

[1013] The user inputs their opinion by voice. For example, they might say, "I feel like there's been a lack of communication within the company recently." The device collects this voice in real time and converts it into text data using a voice recognition function. It also uses an emotion engine to recognize the user's emotions. For example, emotional data such as "dissatisfaction" or "stress" is obtained.

[1014] Sending and receiving text data and emotion data

[1015] The device sends the collected text data and emotion data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and emotion data and stores them in a database. For example, text data such as "I feel that there has been a lack of communication within the company recently" and emotion data such as "dissatisfied" are stored.

[1016] Server analysis and summarization

[1017] The server analyzes the received text data and emotion data. It uses a natural language processing (NLP) engine to extract important opinions and key phrases from the text data, and incorporates the emotion data into the analysis. When similar opinions are collected from multiple users, they are summarized and the main topics are extracted along with the emotion data. For example, a topic such as "lack of internal communication" and the associated emotion data of "dissatisfaction" are extracted.

[1018] Proposing improvement measures based on emotion data

[1019] The server then proposes improvement measures based on the extracted topics and emotion data, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" may be proposed. The server takes emotion data into account to generate improvement measures that better reflect the user's emotions.

[1020] Report preparation and notification

[1021] The server compiles the proposed improvements into a report. The report is a structured document that includes a list of improvements and detailed explanations, as well as analysis results based on the emotion data. The report is then sent back to the device, where it is displayed via a user interface for users and administrators to view. The device also notifies users of the completion of the report via a notification function.

[1022] Specific examples

[1023] For example, if an employee says, "The project is behind schedule, so I'd like to have more frequent meetings," and expresses "dissatisfaction," the following process will be executed:

[1024] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[1025] 2. The device collects this speech and converts it into text, such as, "The project is behind schedule, so please hold more frequent meetings." The emotion engine recognizes the emotion "dissatisfied."

[1026] 3. The device sends the converted text data and emotion data to the server.

[1027] 4. The server receives the text data and emotion data and summarizes it as "increase the frequency of meetings to accelerate project progress." The emotion data "dissatisfied" is also saved.

[1028] 5. The server uses the summary data and emotion data to propose improvement measures, such as "increasing the frequency of meetings from once a week to twice a week." In this case, more detailed measures are proposed, taking into account the emotion of "dissatisfaction."

[1029] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[1030] 7. The terminal notifies the user of the report, and the user can check the report.

[1031] In this way, the present invention realizes a system that carries out a series of processes from collecting voice opinions to proposing improvement measures and creating reports, and by incorporating emotional data, provides more effective measures to improve employee satisfaction.

[1032] The processing flow will be explained below.

[1033] Step 1:

[1034] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[1035] Step 2:

[1036] The device collects the user's voice. The voice input device (such as a microphone) captures the voice data. At this point, it is saved as an audio file.

[1037] Step 3:

[1038] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, the generated text might say, "I feel like there's been a lack of communication within the company recently." The emotion engine then analyzes the voice data and recognizes the user's emotion (for example, "dissatisfied").

[1039] Step 4:

[1040] The device sends the generated text data and emotion data to the server using an HTTP request or the WebSocket communication protocol.

[1041] Step 5:

[1042] The server receives the transmitted text data and emotion data and stores them in a database. For example, the text "There has been a lack of communication within the company recently" and the emotion data "dissatisfied" are stored.

[1043] Step 6:

[1044] The server analyzes the text and sentiment data received from the database, and uses a natural language processing (NLP) engine to extract important opinions and key phrases, which are then analyzed along with the sentiment data.

[1045] Step 7:

[1046] The server summarizes the results of the analysis. If there are multiple opinions, common themes are extracted and a short summary is made, taking into account the emotion data. For example, the topic "lack of internal communication" and the emotion "dissatisfaction" are summarized.

[1047] Step 8:

[1048] The server proposes improvement measures based on the summarized data and emotion data. Specific measures that take the user's emotions into account are generated based on pre-learned data and past performance. For example, it may suggest "holding regular all-employee meetings" or "introducing specialized communication tools."

[1049] Step 9:

[1050] The server compiles the proposed improvement measures in a report format. A report document is created that includes a list of the improvement measures, detailed explanations, and the designation of the departments and business operations that will be affected. The report also includes the results of the analysis based on the emotion data.

[1051] Step 10:

[1052] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[1053] Step 11:

[1054] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[1055] Step 12:

[1056] The device notifies the user that the report is complete via the notification API.

[1057] Through these detailed steps, we have created a system that efficiently collects users' voice opinions, analyzes them, takes into account emotional data, proposes improvement measures, and finally notifies them in the form of a report.

[1058] Example 2

[1059] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1060] In today's work environment, lack of internal communication and declining employee satisfaction are major issues. There is a need for an effective system to efficiently collect employee complaints and opinions and propose appropriate improvement measures. However, traditional manual methods of collecting opinions and proposing improvement measures are time-consuming and labor-intensive, and have the problem of not accurately reflecting employee sentiment.

[1061] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1062] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data and emotion data to the server, means for analyzing and summarizing the received text data and emotion data, means for proposing improvement measures based on the analyzed text data and emotion data, and means for compiling the proposed improvement measures into a report and transmitting it to the terminal. This makes it possible to quickly and accurately collect and analyze employee opinions and emotions, propose optimal improvement measures, and improve employee satisfaction and work efficiency.

[1063] "User" refers to a subject who inputs opinions into the system by voice.

[1064] "Voice input" refers to the process of collecting information by a user speaking into a microphone.

[1065] "Opinions" refers to opinions such as evaluations, suggestions, and impressions expressed by users regarding the system.

[1066] "Device" refers to a device that collects voice and converts it into text data, such as a smartphone or computer.

[1067] "Speech recognition software" refers to an application for converting voice data into text data.

[1068] "Emotion engine" refers to software or algorithms for analyzing emotions from a user's voice.

[1069] "Text data" refers to textual information that expresses a user's opinion and is converted by voice recognition software.

[1070] "Emotion data" refers to information that represents the user's emotional state as analyzed by the emotion engine.

[1071] "Server" refers to a computer system that receives, analyzes, stores, and processes data as needed.

[1072] "Data transmission means" refers to means for transmitting text data and emotion data from a terminal to a server.

[1073] The "HTTP protocol" refers to a communication method for sending and receiving data on the web.

[1074] "WebSocket" refers to a protocol that enables two-way communication.

[1075] "Database" refers to a system that allows a server to store and manage data in a structured manner.

[1076] A "natural language processing engine" refers to technology for analyzing text data and extracting important key phrases and opinions.

[1077] "Keyphrases" refer to words or phrases that are considered particularly important in opinion or text data.

[1078] "Summarizing" refers to the process of combining multiple opinions or data into a concise form.

[1079] "Topics" refer to major themes or topics extracted from the analyzed data.

[1080] "Improvement measures" refer to specific countermeasures proposed based on the analysis results and emotional data.

[1081] "Report" refers to a document summarizing proposed improvements and analysis results.

[1082] "Notification" refers to the act of informing a user of a report or important information.

[1083] This system collects opinions entered by users through speech, converts them into text data, analyzes and summarizes them, proposes improvement measures based on the emotional data, and compiles the results into a report. This system uses common devices and software.

[1084] 1. User voice input and collection

[1085] Users input their opinions by voice using devices such as smartphones or PCs. For example, a user might say, "I feel like there's been a lack of communication within the company recently." At this time, the device activates its microphone and collects voice data. The collected voice data is temporarily stored in a buffer.

[1086] 2. Audio Data Processing

[1087] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert voice data into text data in real time. The converted text data is temporarily stored on the device. At the same time, an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the user's emotions from the voice data and extracts emotional data such as "dissatisfaction" or "stress."

[1088] 3. Sending text data and emotion data

[1089] The device sends text data and emotion data to the server using the HTTP protocol. The data is converted to JSON format and sent to the server along with the user ID and a timestamp.

[1090] 4. Processing of Data by the Server

[1091] The server stores the received text data and sentiment data in a database (e.g., MySQL). The stored data is used as raw material for analysis. The server then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy or NLTK) to extract key phrases and important opinions.

[1092] 5. Proposing improvement measures based on emotion data

[1093] Based on the analysis results, the server uses a pre-trained generative AI model (e.g., GPT-4) to generate specific improvement measures. For example, it may suggest improvement measures such as "holding regular all-employee meetings" or "introducing specialized communication tools." In doing so, it takes into account emotional data to provide improvement measures that better reflect the user's emotions.

[1094] Example prompt: "The user expressed dissatisfaction by stating, 'I feel like our internal communication has been lacking lately.' Please suggest the best way to improve this."

[1095] 6. Preparation and notification of reports

[1096] The server compiles the improvement measures into a report format and structures the analysis results based on the text data, details of the improvement measures, and sentiment data into a document, which can be exported in formats such as PDF or HTML and sent back to the device.

[1097] The terminal displays the received report through a user interface for the user or administrator to view, and also notifies the user that the report has been completed through a notification function.

[1098] This system makes it possible to efficiently and accurately collect employee complaints and opinions and quickly propose improvement measures appropriate to the situation, which is expected to improve employee satisfaction and work efficiency.

[1099] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1100] Step 1:

[1101] The user inputs their opinion by voice. For example, the user might say, "I feel like there's been a lack of internal communication recently." This becomes the input to the system. The device activates the microphone and collects voice data in real time. The collected voice data is temporarily stored in a buffer. The input voice data becomes the output of this step.

[1102] Step 2:

[1103] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data. It also uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state from the voice data. The input here is the voice data collected in step 1, and the output is text data and emotion data. Specifically, the voice data is first input into the software, converted into text data, and emotion analysis is performed in parallel.

[1104] Step 3:

[1105] The device sends text data and emotion data to the server using the HTTP protocol or WebSocket. The input data is the text data and emotion data generated in step 2. These data are converted into JSON format and sent to the server along with the user ID and timestamp. The output is a confirmation of successful transmission. The specific operation includes the process in which the device converts the data into an appropriate format and sends it over the network.

[1106] Step 4:

[1107] The server stores the received text data and emotion data in a database (e.g., MySQL). The input is the data sent in step 3, and after receiving it, it stores it in the database using an SQL statement. The output is a confirmation that the data was successfully saved to the database. Specifically, the server first establishes a database connection and inserts the received data into the appropriate tables.

[1108] Step 5:

[1109] The server analyzes the stored text data and sentiment data using a natural language processing (NLP) engine (e.g., spaCy or NLTK). The input data is the data stored in the database in step 4, and important key phrases and opinions are extracted from it. The output is the analyzed text data, key phrases, and topics. Specifically, the server reads the data from the database, inputs it into the NLP engine, and generates the analysis results.

[1110] Step 6:

[1111] The server uses a generative AI model (e.g., GPT-4) to propose improvement measures based on the analysis results. The input prompt is the analysis result obtained in step 5, and the output is the generated improvement measures. Specifically, the server inputs the prompt into the generative AI model and compiles the obtained output as the optimal improvement measures.

[1112] Step 7:

[1113] The server compiles the proposed improvements into a report and exports it in PDF or HTML format. The input is the improvements obtained in step 6, along with the necessary text data and emotion data. The output is the completed report. Specifically, the server uses a report generation tool to compile and format all the information and export it as a file.

[1114] Step 8:

[1115] The server sends the completed report back to the terminal. The input data is the report created in step 7. This is sent to the terminal as an HTTP response, and the user is notified. The output is a notification that the report was successfully sent and a notification to the user. Specifically, the server sends the report as an HTTP request, and the terminal receives it, displays it on the user interface, and notifies the user.

[1116] (Application example 2)

[1117] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1118] In current virtual stores, it is difficult to quickly and accurately collect user feedback and propose appropriate improvement measures based on it. Furthermore, there is a lack of systems that can provide higher-quality service by recognizing, analyzing, and reflecting emotions in user voice. Therefore, there is a need for a system that can collect voice feedback, convert it into text, analyze emotional data, and propose effective improvement measures in a single flow.

[1119] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data and emotional data, and means for collecting users' voice opinions in a virtual store and proposing improvement measures for products based on the emotional data. This makes it possible to instantly collect users' voice opinions and, based on them, to propose appropriate improvement measures that reflect the emotional data.

[1120] A "user" is an entity that uses the system to input voice opinions.

[1121] "Voice feedback" refers to feedback, requests, opinions, etc. that a user provides to the system by voice.

[1122] The "collection means" refers to a mechanism for detecting the user's voice opinions and capturing them as data.

[1123] "Text data" refers to digital data that has been converted from voice opinions into text information.

[1124] "Emotion data" refers to data containing emotion information extracted from a user's voice opinion.

[1125] "Server" refers to a computer system that analyzes collected text data and emotion data and proposes improvement measures.

[1126] "Analysis means" refers to the process by which the server processes text data and emotion data to extract important information and emotions.

[1127] "Summarization" refers to the process of extracting key points from the analyzed data and summarizing them concisely.

[1128] "Means for proposing improvements" refers to the process by which the server generates specific improvement proposals based on the analysis data.

[1129] "Report" refers to a written summary of proposed improvements and analysis results.

[1130] "Notification" refers to the process by which a user is notified of report completion or specific actions.

[1131] "Virtual store" refers to a virtual shopping environment that offers products and services online and is accessible to users.

[1132] "Presentation" refers to the process by which the system displays and provides information and suggestions to the user.

[1133] 1. System Configuration

[1134] This invention is a system based on collecting opinions input by voice from users and converting them into text data and emotion data. The system is composed of a voice recognition unit, emotion recognition unit, data transmission unit, server unit, analysis unit, summarization unit, proposal unit, report generation unit, and notification unit.

[1135] 2. Voice input and collection

[1136] Users use devices such as smart glasses or head-mounted displays to input their opinions by voice. The device collects the voice via a microphone and converts it into text data in real time. Voice recognition software such as the Google Speech Recognition API is used for voice recognition. Emotion recognition uses natural language processing using the BERT model (for example, "cl-tohoku / bert-base-japanese") to extract the user's emotions.

[1137] 3. Transmission of text data and emotional data

[1138] The device sends the collected text data and emotion data to a server using standard data transmission methods such as the HTTP protocol or WebSocket.

[1139] 4. Server Analysis and Summarization

[1140] The server receives the text and emotion data and stores them in a database. It then uses a natural language processing (NLP) engine to analyze the text data and extract important opinions and key phrases. It also summarizes similar opinions from multiple users and extracts key topics. The summaries reflect the emotion data.

[1141] 5. Propose improvements and generate reports

[1142] The server then uses the extracted topics and sentiment data to propose improvement measures, taking into account pre-learned data and past performance. For example, in response to the comment "the price is high," specific improvement measures such as "holding a sale" or "offering a special offer" are proposed. In this process, the server takes into account sentiment data to generate improvement measures that better reflect the user's sentiment. Finally, the server compiles the proposed improvement measures into a report and sends it to the device. Users and administrators can view the report through the user interface.

[1143] 6. Specific Examples

[1144] If a user says "This product is too expensive" in a virtual store and the emotion "unhappy" is recognized, the following process is executed:

[1145] 1. Users express their opinions.

[1146] 2. The device collects the voice and converts it into text. It then performs emotion recognition and obtains the emotion data of "dissatisfied."

[1147] 3. The device sends the text data and emotion data to the server.

[1148] 4. The server analyzes the received data and summarizes it as "the price is high." It also saves the emotional data "dissatisfied."

[1149] 5. The server then proposes improvements, such as "holding a sale," based on the data, taking into account the emotional data.

[1150] 6. The server generates a report and sends it to the device.

[1151] 7. The terminal notifies the user that the report is complete, and the user can review the report.

[1152] Prompt Sentence Examples

[1153] If a user in your virtual store says "this item is too expensive" about a product, and the system recognizes this as a dissatisfied sentiment, how would you adjust the price or suggest a special offer?

[1154] The above is a specific embodiment of the invention.

[1155] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1156] Step 1:

[1157] The user inputs their opinion by voice.

[1158] Specific actions: The user speaks their opinion into a microphone attached to smart glasses or a head-mounted display. For example, they say, "This product is too expensive."

[1159] Input: Audio data

[1160] Output: Collecting audio data

[1161] Step 2:

[1162] The device collects voice opinions and converts them into text data.

[1163] Specific operation: The device converts the collected voice data into text data in real time using the Google Speech Recognition API. For example, the voice data is converted into text data such as "This product is too expensive."

[1164] Input: Audio data

[1165] Output: Text data

[1166] Step 3:

[1167] The terminal acquires emotion data from the text data.

[1168] Specific operation: The device inputs the converted text data into an emotion recognition model such as "cl-tohoku / bert-base-japanese" and performs emotion analysis. For example, from the text data "This product is too expensive," it obtains emotion data such as "dissatisfied."

[1169] Input: Text data

[1170] Output: Emotion data

[1171] Step 4:

[1172] The terminal transmits the text data and the emotion data to the server.

[1173] Specific operation: The device sends text data and emotion data to the server using the HTTP protocol. For example, the text data "This product is too expensive" and the emotion data "Dissatisfied" are sent.

[1174] Input: Text data, emotion data

[1175] Output: Send data to the server

[1176] Step 5:

[1177] The server analyzes the received text data and emotion data and stores them in a database.

[1178] Specific operation: The server analyzes the received data and extracts key phrases using a natural language processing (NLP) engine. It also stores the text data and emotion data in a database. For example, it stores the text data "This product is too expensive" and the emotion data "Dissatisfied."

[1179] Input: Text data, emotion data

[1180] Output: Analysis results, saved in database

[1181] Step 6:

[1182] The server analyzes multiple opinions and performs summarization and topic extraction.

[1183] Specific operation: The server analyzes the received text data and summarizes important opinions and key phrases. It also extracts key topics based on the sentiment data. For example, the topic "expensive" and the sentiment "dissatisfied" are extracted.

[1184] Input: Text data, emotion data

[1185] Output: Summary data, topics

[1186] Step 7:

[1187] The server proposes improvement measures based on the summary data and emotion data.

[1188] Specific actions: The server uses the data it has learned in advance and past performance data to generate specific improvement measures from the analysis data. For example, it may suggest things like "run a sale" or "offer a special offer."

[1189] Input: Summary data, emotion data

[1190] Output: Improvements

[1191] Step 8:

[1192] The server compiles improvement measures into a report and sends it to the terminal.

[1193] Specific operation: The server compiles the proposed improvement measures into a report format and sends it to the terminal using the HTTP protocol. For example, an improvement measure such as "conducting a sale" is included in the report and sent to the terminal.

[1194] Input: Remedy

[1195] Output: Send report

[1196] Step 9:

[1197] The terminal receives the report and displays and notifies it through a user interface.

[1198] Specific operation: The device displays the received report on the user interface and notifies the user of the completion of the report through a notification function. For example, a notification saying "New improvement measures have been proposed" is displayed on the user's device.

[1199] Input: report

[1200] Output: report display, notification

[1201] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1202] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1203] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1204] [Fourth embodiment]

[1205] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1206] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1207] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1208] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1209] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1210] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1211] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1212] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1213] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1214] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1215] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1216] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1217] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1218] Overall system overview

[1219] This invention is a system that collects opinions entered by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. Specifically, users enter their opinions by voice using a terminal, and the terminal collects and converts the voice data into text. The text data is then sent to a server, where it is analyzed and a summary and improvement proposals are generated. Finally, the text data is sent to the terminal as a report and notified to the user or administrator.

[1220] Program processing overview

[1221] User voice input and collection

[1222] The user can input their opinion by voice. For example, they can say, "I feel like there's been a lack of internal communication recently." The device collects this voice in real time and converts it into text data using its voice recognition function.

[1223] Sending and receiving text data

[1224] The device sends the collected text data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and stores it in a database. For example, the text data stored might read, "I feel like there's been a lack of internal communication recently."

[1225] Server analysis and summarization

[1226] The server analyzes the received text data and uses natural language processing (NLP) algorithms to extract important opinions and key phrases. When similar opinions are collected from multiple users, these are summarized to extract major topics. For example, a topic such as "lack of internal communication" may be extracted.

[1227] Proposal for improvement

[1228] Based on the extracted topics, the server proposes improvement measures, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" are proposed.

[1229] Report preparation and notification

[1230] The server compiles the proposed improvements into a report, which is then sent back to the terminal. The terminal displays the received report through a user interface so that users and administrators can view it. Furthermore, the terminal notifies the user when the report is complete through a notification function.

[1231] Specific examples

[1232] For example, if an employee says, "We're behind on our project, so we'd like to have more frequent meetings," the following process will be executed:

[1233] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[1234] 2. The device collects this voice and converts it into text that reads, "The project is progressing slowly, so please hold more frequent meetings."

[1235] 3. The device sends the converted text data to the server.

[1236] 4. The server receives the text data and summarizes it as "increase the frequency of meetings to accelerate project progress."

[1237] 5. Based on the summary data, the server suggests improvement measures such as "increase the frequency of meetings from once a week to twice a week."

[1238] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[1239] 7. The terminal notifies the user of the report, and the user can check the report.

[1240] In this way, the present invention provides a system that performs a series of processes from collecting voice opinions to proposing improvement measures and creating a report.

[1241] The processing flow will be explained below.

[1242] Step 1:

[1243] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[1244] Step 2:

[1245] The terminal collects the user's voice using a voice input device such as a microphone.

[1246] Step 3:

[1247] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, it generates text such as "I feel that internal communication has been lacking recently."

[1248] Step 4:

[1249] The device sends the generated text data to the server via an HTTP request or WebSocket.

[1250] Step 5:

[1251] The server receives the text data and stores it in a database, managed in an appropriate format such as SQL or NoSQL.

[1252] Step 6:

[1253] The server analyzes the received text data and uses a natural language processing (NLP) engine to extract important opinions and key phrases.

[1254] Step 7:

[1255] The server summarizes the analysis results. If there are multiple opinions, common themes are extracted and summarized briefly.

[1256] Step 8:

[1257] The server proposes improvement measures based on the summarized data, and generates specific measures by referencing past data and machine learning models.

[1258] Step 9:

[1259] The server will compile the proposed improvements into a report, creating a structured document that lists the improvements and includes detailed explanations.

[1260] Step 10:

[1261] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[1262] Step 11:

[1263] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[1264] Step 12:

[1265] The device notifies the user when the report is complete. The notification content is determined based on business logic via the notification API.

[1266] Through this series of steps, we will create a system that can efficiently collect and analyze user opinions and provide appropriate improvement measures in real time.

[1267] Example 1

[1268] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1269] Conventional opinion collection systems lack the ability to effectively convert user voice input into text data, analyze it, and summarize it, resulting in problems that do not lead to improved productivity. Furthermore, the process of proposing improvement measures is not automated, which consumes a lot of time and effort from users, and the process of proposing improvement measures lacks consistency. Considering these issues, there is a need for an efficient and consistent opinion collection and improvement proposal system.

[1270] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1271] In this invention, the server includes means for analyzing received text data using a natural language processing algorithm and extracting key phrases and major topics, means for proposing improvement measures using a generative AI model, and means for compiling the proposed improvement measures into a report and sending it to the terminal. This makes it possible to efficiently convert user speech into text data, extract important opinions and topics, and automatically provide consistent improvement measures.

[1272] A "user" is a person who uses the system to provide voice input and opinions and feedback.

[1273] A "terminal" is a device that allows a user to input voice and has the function of collecting voice and converting it into text data.

[1274] The "server" is a central processing unit that receives and analyzes text data sent from the terminal, proposes improvements, and compiles them into a report.

[1275] "Voice input" refers to the act of a user giving voice to a terminal to give opinions or feedback.

[1276] "Text data" is data in the form of a character string converted based on voice input.

[1277] A "natural language processing algorithm" is a programming technique for analyzing text data and extracting key phrases and major topics.

[1278] A "generative AI model" is an artificial intelligence program that generates new information and suggestions based on pre-trained data.

[1279] "Improvement measures" are specific responses or countermeasures to the extracted opinions or topics.

[1280] A "report" is a document summarizing proposed improvements and is provided to users and administrators.

[1281] "User interface" refers to the display screen and operation means that allow a user to check reports and notifications on a terminal.

[1282] "Notifications" are messages or signals that notify the user of report completion or information from the system.

[1283] The present invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system uses the following hardware and software:

[1284] System configuration

[1285] User voice input and collection

[1286] Users input their opinions by voice using a device such as a smartphone, PC, or tablet. For example, a user might say, "I feel like there's been a lack of internal communication lately." The device collects the voice through a built-in or connected microphone and converts it into text data in real time using the Google Cloud Speech-to-Text API.

[1287] Sending and receiving text data

[1288] The device sends the converted text data to the server. This communication uses standard data transmission methods such as the HTTP protocol or WebSocket. The server stores the received text data in a database (for example, MySQL). As a specific example, the text data "I feel that there has been a lack of internal communication recently" is sent to the server and stored in the database.

[1289] Server analysis and summarization

[1290] The text data received by the server is analyzed using natural language processing (NLP) algorithms, such as NLTK or spaCy. Important opinions and key phrases are extracted, and similar opinions from multiple users are summarized. For example, a topic such as "lack of internal communication" is extracted.

[1291] Proposal for improvement

[1292] Based on the extracted topics, the server uses a generative AI model (such as GPT-4) to propose improvement measures. The server uses pre-trained data and past performance data to generate specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[1293] Report preparation and notification

[1294] The server compiles the proposed improvements into a report and sends it to the device. The created report is then sent back to the device, which displays the received report through a user interface (for example, an interface using React.js). In addition, the device notifies the user of the completion of the report through a notification function.

[1295] Specific examples

[1296] For example, if an employee says, "The project is progressing slowly, so I would like to have more frequent meetings," the process will proceed as follows:

[1297] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[1298] 2. The device collects this voice and converts it into text, such as "The project is behind schedule, so please hold more frequent meetings." This is done using the Google Cloud Speech-to-Text API.

[1299] 3. The terminal sends the converted text data to the server using the HTTP protocol.

[1300] 4. The server receives the text data and summarizes it as "Increase the frequency of meetings to accelerate project progress." NLTK and spaCy are used for analysis.

[1301] 5. The server uses the summary data to suggest improvements, such as increasing the frequency of meetings from once a week to twice a week. It uses the generative AI model GPT-4.

[1302] 6. The server compiles the proposed improvements into a report and sends it to the device. React.js is used for the front end.

[1303] 7. The terminal notifies the user of the report, and the user can check the report.

[1304] In this way, the present invention provides a system that efficiently executes a series of processes from collecting voice feedback to proposing improvement measures and creating reports, thereby improving productivity and providing consistent improvement measures.

[1305] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1306] Step 1:

[1307] The user inputs their opinion by voice.

[1308] Specifically, the user speaks to their smartphone or PC, saying, "I feel like there's been a lack of communication within the company recently." This voice is then input into the device.

[1309] Step 2:

[1310] The device collects the audio.

[1311] Specifically, the device's microphone captures the audio and stores it as audio data. This stored audio data is used as input in the next step.

[1312] Step 3:

[1313] The device converts the voice data into text data using the Google Cloud Speech-to-Text API.

[1314] Specifically, the voice data is sent to the API, and a speech recognition algorithm is applied, generating text data such as "I feel that internal communication has been lacking recently."

[1315] Step 4:

[1316] The terminal transmits the converted text data to the server.

[1317] Specifically, the device generates an HTTP POST request and sends it to the server's API endpoint, including the text data as a payload. This HTTP request is input, and the text data is sent to the server.

[1318] Step 5:

[1319] The server receives the text data and stores it in a database.

[1320] Specifically, the server receives an HTTP request and extracts the text data from the payload. This text data is input and stored in a database. The stored text data is used as output in the next step.

[1321] Step 6:

[1322] The server analyzes the text data using natural language processing algorithms.

[1323] Specifically, the server uses NLTK and spaCy to extract key phrases and major topics. This analysis takes text data as input and outputs a list of key phrases and topics. For example, the topic "lack of internal communication" might be extracted.

[1324] Step 7:

[1325] The server uses a generative AI model based on the extracted topics to propose improvement measures.

[1326] Specifically, the server inputs topic data into a generative AI model (such as GPT-4), which generates improvement measures based on pre-trained data. This process generates specific improvement measures as output, such as "holding regular all-employee meetings" and "introducing specialized communication tools."

[1327] Step 8:

[1328] The server compiles the proposed improvements in a report format.

[1329] Specifically, the proposed improvements are formatted as a text document and compiled into a report. The report generated in this step is the output.

[1330] Step 9:

[1331] The server sends the report to the terminal.

[1332] Specifically, the report is attached to the HTTP response and sent to the terminal. During this process, the report becomes input and is sent to the terminal.

[1333] Step 10:

[1334] The terminal receives the report and displays it through a user interface.

[1335] Specifically, the report received by the device is displayed on the UI so that the user can view it. The report is displayed using an interface that uses React.js. The report is input and displayed on the user interface.

[1336] Step 11:

[1337] The terminal notifies the user when the report is complete.

[1338] Specifically, the notification function is used to send a pop-up or email notification to the user to let them know that the report is complete. This allows the user to be aware of the existence of the report. The notification is used as input to notify the user.

[1339] (Application example 1)

[1340] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1341] In conventional content distribution services, the challenge is to quickly and accurately collect user feedback and propose specific improvement measures to improve the quality of the service based on that feedback. In particular, there is a need to provide an automated process from collecting voice opinions to analyzing them, proposing improvement measures, and creating reports.

[1342] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1343] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data to the server, means for analyzing and summarizing the text data received by the server, means for proposing improvements based on the analyzed data by the server, means for compiling the proposed improvements as a report and transmitting it to the terminal, and means for collecting and analyzing users' voice opinions and proposing improvements to improve the quality of the service in a content distribution service, thereby enabling high-quality content distribution that quickly reflects user opinions.

[1344] "User" refers to a user who uses the content distribution service and inputs their opinion by voice.

[1345] "Voice input" refers to a user verbally voicing opinions or feedback.

[1346] "Text data" refers to digital data converted from voice input into text information.

[1347] "Server" refers to a computer system that communicates over a network and analyzes, summarizes, and creates reports on received data.

[1348] "Analysis" refers to the process of understanding and classifying received text data using techniques such as natural language processing, and extracting important information.

[1349] "Summarization" refers to the process of summarizing the information obtained through analysis in a concise and easy-to-understand manner.

[1350] "Improvement measures" refer to specific solutions for improving the quality of content distribution services based on collected user opinions.

[1351] "Report" means a document that provides a summary and recommendations for improvement to an administrator or content creator.

[1352] "Terminal" refers to hardware devices that process information, such as computers and smartphones used by users and administrators.

[1353] "Content distribution service" refers to a system that provides digital content such as videos and music via a network.

[1354] "Feedback" refers to opinions and impressions that users give regarding the content distribution service.

[1355] The present invention is a system for collecting user feedback in the form of voice in a content distribution service, analyzing and summarizing the feedback, and proposing and notifying improvement measures. The following describes an embodiment of the present invention.

[1356] Overall system overview

[1357] This system consists of a series of processes to collect users' voice opinions, convert them into text data, analyze them, summarize them, and propose improvements. Specifically, the main components are the device used by the user, voice recognition software, a server that manages and analyzes the text data, and a generative AI model that proposes improvements.

[1358] Hardware Configuration

[1359] Device: A user device such as a smartphone, tablet, or computer.

[1360] Microphone: Used for voice input.

[1361] Server: A high-performance computer system that runs in the cloud or on-premise.

[1362] Software Configuration

[1363] Speech Recognition Library: Uses Python's speech_recognition library to convert speech to text.

[1364] HTTP communication library: Use the requests library to send the converted speech to text data to the server.

[1365] Natural Language Processing Library: Utilizes the transformers library to analyze text data and generate summaries and remediation suggestions.

[1366] Generative AI model: Processes prompt sentences to generate improvements using models such as GPT-3.

[1367] Specific operation example

[1368] 1. Voice input and collection

[1369] Users can provide feedback to the content distribution service by voice, such as, "The subtitles for this video are displayed late, so I'd like them to be fixed."

[1370] 2. Speech-to-text

[1371] The device uses the speech_recognition library to collect the user's voice in real time and convert it into text data. The voice input data is temporarily stored in the device's memory while the conversion process is performed.

[1372] 3. Sending and receiving text data

[1373] The terminal uses the requests library to send the converted text data to the server via the HTTP protocol, where it is stored and prepared for analysis.

[1374] 4. Server Analysis and Summarization

[1375] The server analyzes the received text data using the transformers library to extract important opinions and key phrases, and generates a summary based on the analysis results, concisely summarizing the main points the user wants to make.

[1376] 5. Proposal for improvement

[1377] The server uses a summary of the text data to suggest improvement measures, taking into account pre-trained data and past performance. The server uses a generative AI model such as GPT-3 to generate the improvement measures by inputting the following prompt sentence:

[1378] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[1379] 6. Preparation and notification of reports

[1380] The server compiles the proposed improvements into a report and sends it to the terminal. The terminal displays the received report through a user interface so that users and administrators can easily check it. It also notifies the user when the report is complete through a notification function.

[1381] The present invention makes it possible to quickly reflect user opinions and deliver high-quality content.

[1382] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1383] Step 1:

[1384] User voice input

[1385] A user inputs their opinion to a content distribution service by voice, for example, "The subtitles for this video are displayed late, so I would like them to be fixed." The user's voice is collected as input, and the output is audio data. Specifically, the system collects the voice in real time using the device's microphone.

[1386] Step 2:

[1387] Converting voice to text data

[1388] The device uses the speech_recognition library to convert speech into text data. The collected speech data is used as input. For data processing, the speech recognition algorithm analyzes the speech data and generates corresponding text. The output is text data. Specifically, the speech recognition software converts speech data into text data.

[1389] Step 3:

[1390] Send text data to the server

[1391] The terminal sends the converted text data to the server using the requests library. The text data generated earlier is used as input. Data is sent using the HTTP protocol. The output is text data stored on the server. Specifically, an HTTP request is sent to a specific endpoint on the server.

[1392] Step 4:

[1393] Server-based analysis and summarization of text data

[1394] The server analyzes and summarizes the received text data using the transformers library. The text data stored on the server is used as input. To process the data, a natural language processing algorithm analyzes the text, extracts important key phrases and opinions, and generates a summary. The output is the summarized text. Specifically, the natural language processing model analyzes the text data and extracts important information.

[1395] Step 5:

[1396] Server-suggested improvement measures

[1397] The server proposes improvement measures based on the summarized text data, taking into account pre-learned data and past performance. The summarized text is used as input. As a data calculation, a generative AI model (e.g., GPT-3) generates improvement measures based on the prompt sentence. The output is the proposed improvement measures. Specifically, the generative AI model operates using the following prompt sentence:

[1398] "Please suggest improvements based on the feedback below: Feedback: The subtitles for this video are delayed and should be fixed."

[1399] Step 6:

[1400] Report preparation and notification

[1401] The server creates a report including the proposed improvements and sends it to the terminal. The user receives a notification and can view the report. The proposed improvements are used as input. Data processing involves formatting the improvements into a report. The output is the report sent to the terminal and the associated notification. Specifically, the report is formatted in JSON or another format and sent to the terminal using the HTTP protocol. The terminal displays the received report through a user interface and notifies the user.

[1402] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1403] Overall system overview

[1404] This invention is a system that collects opinions input by voice from users, converts them into text data, analyzes and summarizes them, and proposes improvements. This system incorporates an emotion engine that recognizes the user's emotions from the voice input and reflects them in the analysis and proposed improvements.

[1405] Program processing overview

[1406] User voice input and collection

[1407] The user inputs their opinion by voice. For example, they might say, "I feel like there's been a lack of communication within the company recently." The device collects this voice in real time and converts it into text data using a voice recognition function. It also uses an emotion engine to recognize the user's emotions. For example, emotional data such as "dissatisfaction" or "stress" is obtained.

[1408] Sending and receiving text data and emotion data

[1409] The device sends the collected text data and emotion data to a server. This communication is carried out using standard data transmission methods such as the HTTP protocol or WebSocket. The server receives the text data and emotion data and stores them in a database. For example, text data such as "I feel that there has been a lack of communication within the company recently" and emotion data such as "dissatisfied" are stored.

[1410] Server analysis and summarization

[1411] The server analyzes the received text data and emotion data. It uses a natural language processing (NLP) engine to extract important opinions and key phrases from the text data, and incorporates the emotion data into the analysis. When similar opinions are collected from multiple users, they are summarized and the main topics are extracted along with the emotion data. For example, a topic such as "lack of internal communication" and the associated emotion data of "dissatisfaction" are extracted.

[1412] Proposing improvement measures based on emotion data

[1413] The server then proposes improvement measures based on the extracted topics and emotion data, taking into account pre-learned data and past performance data. For example, specific improvement measures such as "holding regular all-employee meetings" and "introducing specialized communication tools" may be proposed. The server takes emotion data into account to generate improvement measures that better reflect the user's emotions.

[1414] Report preparation and notification

[1415] The server compiles the proposed improvements into a report. The report is a structured document that includes a list of improvements and detailed explanations, as well as analysis results based on the emotion data. The report is then sent back to the device, where it is displayed via a user interface for users and administrators to view. The device also notifies users of the completion of the report via a notification function.

[1416] Specific examples

[1417] For example, if an employee says, "The project is behind schedule, so I'd like to have more frequent meetings," and expresses "dissatisfaction," the following process will be executed:

[1418] 1. A user says, "The project is progressing slowly, so I would like to have more frequent meetings."

[1419] 2. The device collects this speech and converts it into text, such as, "The project is behind schedule, so please hold more frequent meetings." The emotion engine recognizes the emotion "dissatisfied."

[1420] 3. The device sends the converted text data and emotion data to the server.

[1421] 4. The server receives the text data and emotion data and summarizes it as "increase the frequency of meetings to accelerate project progress." The emotion data "dissatisfied" is also saved.

[1422] 5. The server uses the summary data and emotion data to propose improvement measures, such as "increasing the frequency of meetings from once a week to twice a week." In this case, more detailed measures are proposed, taking into account the emotion of "dissatisfaction."

[1423] 6. The server compiles the proposed improvements into a report and sends it to the terminal.

[1424] 7. The terminal notifies the user of the report, and the user can check the report.

[1425] In this way, the present invention realizes a system that carries out a series of processes from collecting voice opinions to proposing improvement measures and creating reports, and by incorporating emotional data, provides more effective measures to improve employee satisfaction.

[1426] The processing flow will be explained below.

[1427] Step 1:

[1428] The user inputs their opinion by voice. For example, they might say, "I feel that internal communication has been lacking recently."

[1429] Step 2:

[1430] The device collects the user's voice. The voice input device (such as a microphone) captures the voice data. At this point, it is saved as an audio file.

[1431] Step 3:

[1432] The device's voice recognition function analyzes the collected voice data and converts it into text data. For example, the generated text might say, "I feel like there's been a lack of communication within the company recently." The emotion engine then analyzes the voice data and recognizes the user's emotion (for example, "dissatisfied").

[1433] Step 4:

[1434] The device sends the generated text data and emotion data to the server using an HTTP request or the WebSocket communication protocol.

[1435] Step 5:

[1436] The server receives the transmitted text data and emotion data and stores them in a database. For example, the text "There has been a lack of communication within the company recently" and the emotion data "dissatisfied" are stored.

[1437] Step 6:

[1438] The server analyzes the text and sentiment data received from the database, and uses a natural language processing (NLP) engine to extract important opinions and key phrases, which are then analyzed along with the sentiment data.

[1439] Step 7:

[1440] The server summarizes the results of the analysis. If there are multiple opinions, common themes are extracted and a short summary is made, taking into account the emotion data. For example, the topic "lack of internal communication" and the emotion "dissatisfaction" are summarized.

[1441] Step 8:

[1442] The server proposes improvement measures based on the summarized data and emotion data. Specific measures that take the user's emotions into account are generated based on pre-learned data and past performance. For example, it may suggest "holding regular all-employee meetings" or "introducing specialized communication tools."

[1443] Step 9:

[1444] The server compiles the proposed improvement measures in a report format. A report document is created that includes a list of the improvement measures, detailed explanations, and the designation of the departments and business operations that will be affected. The report also includes the results of the analysis based on the emotion data.

[1445] Step 10:

[1446] The server sends the completed report to the device, where it encodes and encrypts the data before sending it to ensure communication security.

[1447] Step 11:

[1448] The terminal displays the received report on the user interface, allowing the user to view the report through the terminal and confirm improvement measures.

[1449] Step 12:

[1450] The device notifies the user that the report is complete via the notification API.

[1451] Through these detailed steps, we have created a system that efficiently collects users' voice opinions, analyzes them, takes into account emotional data, proposes improvement measures, and finally notifies them in the form of a report.

[1452] Example 2

[1453] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1454] In today's work environment, lack of internal communication and declining employee satisfaction are major issues. There is a need for an effective system to efficiently collect employee complaints and opinions and propose appropriate improvement measures. However, traditional manual methods of collecting opinions and proposing improvement measures are time-consuming and labor-intensive, and have the problem of not accurately reflecting employee sentiment.

[1455] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1456] In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data, means for transmitting the text data and emotion data to the server, means for analyzing and summarizing the received text data and emotion data, means for proposing improvement measures based on the analyzed text data and emotion data, and means for compiling the proposed improvement measures into a report and transmitting it to the terminal. This makes it possible to quickly and accurately collect and analyze employee opinions and emotions, propose optimal improvement measures, and improve employee satisfaction and work efficiency.

[1457] "User" refers to a subject who inputs opinions into the system by voice.

[1458] "Voice input" refers to the process of collecting information by a user speaking into a microphone.

[1459] "Opinions" refers to opinions such as evaluations, suggestions, and impressions expressed by users regarding the system.

[1460] "Device" refers to a device that collects voice and converts it into text data, such as a smartphone or computer.

[1461] "Speech recognition software" refers to an application for converting voice data into text data.

[1462] "Emotion engine" refers to software or algorithms for analyzing emotions from a user's voice.

[1463] "Text data" refers to textual information that expresses a user's opinion and is converted by voice recognition software.

[1464] "Emotion data" refers to information that represents the user's emotional state as analyzed by the emotion engine.

[1465] "Server" refers to a computer system that receives, analyzes, stores, and processes data as needed.

[1466] "Data transmission means" refers to means for transmitting text data and emotion data from a terminal to a server.

[1467] The "HTTP protocol" refers to a communication method for sending and receiving data on the web.

[1468] "WebSocket" refers to a protocol that enables two-way communication.

[1469] "Database" refers to a system that allows a server to store and manage data in a structured manner.

[1470] A "natural language processing engine" refers to technology for analyzing text data and extracting important key phrases and opinions.

[1471] "Keyphrases" refer to words or phrases that are considered particularly important in opinion or text data.

[1472] "Summarizing" refers to the process of combining multiple opinions or data into a concise form.

[1473] "Topics" refer to major themes or topics extracted from the analyzed data.

[1474] "Improvement measures" refer to specific countermeasures proposed based on the analysis results and emotional data.

[1475] "Report" refers to a document summarizing proposed improvements and analysis results.

[1476] "Notification" refers to the act of informing a user of a report or important information.

[1477] This system collects opinions entered by users through speech, converts them into text data, analyzes and summarizes them, proposes improvement measures based on the emotional data, and compiles the results into a report. This system uses common devices and software.

[1478] 1. User voice input and collection

[1479] Users input their opinions by voice using devices such as smartphones or PCs. For example, a user might say, "I feel like there's been a lack of communication within the company recently." At this time, the device activates its microphone and collects voice data. The collected voice data is temporarily stored in a buffer.

[1480] 2. Audio Data Processing

[1481] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert voice data into text data in real time. The converted text data is temporarily stored on the device. At the same time, an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the user's emotions from the voice data and extracts emotional data such as "dissatisfaction" or "stress."

[1482] 3. Sending text data and emotion data

[1483] The device sends text data and emotion data to the server using the HTTP protocol. The data is converted to JSON format and sent to the server along with the user ID and a timestamp.

[1484] 4. Processing of Data by the Server

[1485] The server stores the received text data and sentiment data in a database (e.g., MySQL). The stored data is used as raw material for analysis. The server then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy or NLTK) to extract key phrases and important opinions.

[1486] 5. Proposing improvement measures based on emotion data

[1487] Based on the analysis results, the server uses a pre-trained generative AI model (e.g., GPT-4) to generate specific improvement measures. For example, it may suggest improvement measures such as "holding regular all-employee meetings" or "introducing specialized communication tools." In doing so, it takes into account emotional data to provide improvement measures that better reflect the user's emotions.

[1488] Example prompt: "The user expressed dissatisfaction by stating, 'I feel like our internal communication has been lacking lately.' Please suggest the best way to improve this."

[1489] 6. Preparation and notification of reports

[1490] The server compiles the improvement measures into a report format and structures the analysis results based on the text data, details of the improvement measures, and sentiment data into a document, which can be exported in formats such as PDF or HTML and sent back to the device.

[1491] The terminal displays the received report through a user interface for the user or administrator to view, and also notifies the user that the report has been completed through a notification function.

[1492] This system makes it possible to efficiently and accurately collect employee complaints and opinions and quickly propose improvement measures appropriate to the situation, which is expected to improve employee satisfaction and work efficiency.

[1493] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1494] Step 1:

[1495] The user inputs their opinion by voice. For example, the user might say, "I feel like there's been a lack of internal communication recently." This becomes the input to the system. The device activates the microphone and collects voice data in real time. The collected voice data is temporarily stored in a buffer. The input voice data becomes the output of this step.

[1496] Step 2:

[1497] The device uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data. It also uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the user's emotional state from the voice data. The input here is the voice data collected in step 1, and the output is text data and emotion data. Specifically, the voice data is first input into the software, converted into text data, and emotion analysis is performed in parallel.

[1498] Step 3:

[1499] The device sends text data and emotion data to the server using the HTTP protocol or WebSocket. The input data is the text data and emotion data generated in step 2. These data are converted into JSON format and sent to the server along with the user ID and timestamp. The output is a confirmation of successful transmission. The specific operation includes the process in which the device converts the data into an appropriate format and sends it over the network.

[1500] Step 4:

[1501] The server stores the received text data and emotion data in a database (e.g., MySQL). The input is the data sent in step 3, and after receiving it, it stores it in the database using an SQL statement. The output is a confirmation that the data was successfully saved to the database. Specifically, the server first establishes a database connection and inserts the received data into the appropriate tables.

[1502] Step 5:

[1503] The server analyzes the stored text data and sentiment data using a natural language processing (NLP) engine (e.g., spaCy or NLTK). The input data is the data stored in the database in step 4, and important key phrases and opinions are extracted from it. The output is the analyzed text data, key phrases, and topics. Specifically, the server reads the data from the database, inputs it into the NLP engine, and generates the analysis results.

[1504] Step 6:

[1505] The server uses a generative AI model (e.g., GPT-4) to propose improvement measures based on the analysis results. The input prompt is the analysis result obtained in step 5, and the output is the generated improvement measures. Specifically, the server inputs the prompt into the generative AI model and compiles the obtained output as the optimal improvement measures.

[1506] Step 7:

[1507] The server compiles the proposed improvements into a report and exports it in PDF or HTML format. The input is the improvements obtained in step 6, along with the necessary text data and emotion data. The output is the completed report. Specifically, the server uses a report generation tool to compile and format all the information and export it as a file.

[1508] Step 8:

[1509] The server sends the completed report back to the terminal. The input data is the report created in step 7. This is sent to the terminal as an HTTP response, and the user is notified. The output is a notification that the report was successfully sent and a notification to the user. Specifically, the server sends the report as an HTTP request, and the terminal receives it, displays it on the user interface, and notifies the user.

[1510] (Application example 2)

[1511] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1512] In current virtual stores, it is difficult to quickly and accurately collect user feedback and propose appropriate improvement measures based on it. Furthermore, there is a lack of systems that can provide higher-quality service by recognizing, analyzing, and reflecting emotions in user voice. Therefore, there is a need for a system that can collect voice feedback, convert it into text, analyze emotional data, and propose effective improvement measures in a single flow.

[1513] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting opinions input by voice from users, means for converting the collected voice into text data and emotional data, and means for collecting users' voice opinions in a virtual store and proposing improvement measures for products based on the emotional data. This makes it possible to instantly collect users' voice opinions and, based on them, to propose appropriate improvement measures that reflect the emotional data.

[1514] A "user" is an entity that uses the system to input voice opinions.

[1515] "Voice feedback" refers to feedback, requests, opinions, etc. that a user provides to the system by voice.

[1516] The "collection means" refers to a mechanism for detecting the user's voice opinions and capturing them as data.

[1517] "Text data" refers to digital data that has been converted from voice opinions into text information.

[1518] "Emotion data" refers to data containing emotion information extracted from a user's voice opinion.

[1519] "Server" refers to a computer system that analyzes collected text data and emotion data and proposes improvement measures.

[1520] "Analysis means" refers to the process by which the server processes text data and emotion data to extract important information and emotions.

[1521] "Summarization" refers to the process of extracting key points from the analyzed data and summarizing them concisely.

[1522] "Means for proposing improvements" refers to the process by which the server generates specific improvement proposals based on the analysis data.

[1523] "Report" refers to a written summary of proposed improvements and analysis results.

[1524] "Notification" refers to the process by which a user is notified of report completion or specific actions.

[1525] "Virtual store" refers to a virtual shopping environment that offers products and services online and is accessible to users.

[1526] "Presentation" refers to the process by which the system displays and provides information and suggestions to the user.

[1527] 1. System Configuration

[1528] This invention is a system based on collecting opinions input by voice from users and converting them into text data and emotion data. The system is composed of a voice recognition unit, emotion recognition unit, data transmission unit, server unit, analysis unit, summarization unit, proposal unit, report generation unit, and notification unit.

[1529] 2. Voice input and collection

[1530] Users use devices such as smart glasses or head-mounted displays to input their opinions by voice. The device collects the voice via a microphone and converts it into text data in real time. Voice recognition software such as the Google Speech Recognition API is used for voice recognition. Emotion recognition uses natural language processing using the BERT model (for example, "cl-tohoku / bert-base-japanese") to extract the user's emotions.

[1531] 3. Transmission of text data and emotional data

[1532] The device sends the collected text data and emotion data to a server using standard data transmission methods such as the HTTP protocol or WebSocket.

[1533] 4. Server Analysis and Summarization

[1534] The server receives the text and emotion data and stores them in a database. It then uses a natural language processing (NLP) engine to analyze the text data and extract important opinions and key phrases. It also summarizes similar opinions from multiple users and extracts key topics. The summaries reflect the emotion data.

[1535] 5. Propose improvements and generate reports

[1536] The server then uses the extracted topics and sentiment data to propose improvement measures, taking into account pre-learned data and past performance. For example, in response to the comment "the price is high," specific improvement measures such as "holding a sale" or "offering a special offer" are proposed. In this process, the server takes into account sentiment data to generate improvement measures that better reflect the user's sentiment. Finally, the server compiles the proposed improvement measures into a report and sends it to the device. Users and administrators can view the report through the user interface.

[1537] 6. Specific Examples

[1538] If a user says "This product is too expensive" in a virtual store and the emotion "unhappy" is recognized, the following process is executed:

[1539] 1. Users express their opinions.

[1540] 2. The device collects the voice and converts it into text. It then performs emotion recognition and obtains the emotion data of "dissatisfied."

[1541] 3. The device sends the text data and emotion data to the server.

[1542] 4. The server analyzes the received data and summarizes it as "the price is high." It also saves the emotional data "dissatisfied."

[1543] 5. The server then proposes improvements, such as "holding a sale," based on the data, taking into account the emotional data.

[1544] 6. The server generates a report and sends it to the device.

[1545] 7. The terminal notifies the user that the report is complete, and the user can review the report.

[1546] Prompt Sentence Examples

[1547] If a user in your virtual store says "this item is too expensive" about a product, and the system recognizes this as a dissatisfied sentiment, how would you adjust the price or suggest a special offer?

[1548] The above is a specific embodiment of the invention.

[1549] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1550] Step 1:

[1551] The user inputs their opinion by voice.

[1552] Specific actions: The user speaks their opinion into a microphone attached to smart glasses or a head-mounted display. For example, they say, "This product is too expensive."

[1553] Input: Audio data

[1554] Output: Collecting audio data

[1555] Step 2:

[1556] The device collects voice opinions and converts them into text data.

[1557] Specific operation: The device converts the collected voice data into text data in real time using the Google Speech Recognition API. For example, the voice data is converted into text data such as "This product is too expensive."

[1558] Input: Audio data

[1559] Output: Text data

[1560] Step 3:

[1561] The terminal acquires emotion data from the text data.

[1562] Specific operation: The device inputs the converted text data into an emotion recognition model such as "cl-tohoku / bert-base-japanese" and performs emotion analysis. For example, from the text data "This product is too expensive," it obtains emotion data such as "dissatisfied."

[1563] Input: Text data

[1564] Output: Emotion data

[1565] Step 4:

[1566] The terminal transmits the text data and the emotion data to the server.

[1567] Specific operation: The device sends text data and emotion data to the server using the HTTP protocol. For example, the text data "This product is too expensive" and the emotion data "Dissatisfied" are sent.

[1568] Input: Text data, emotion data

[1569] Output: Send data to the server

[1570] Step 5:

[1571] The server analyzes the received text data and emotion data and stores them in a database.

[1572] Specific operation: The server analyzes the received data and extracts key phrases using a natural language processing (NLP) engine. It also stores the text data and emotion data in a database. For example, it stores the text data "This product is too expensive" and the emotion data "Dissatisfied."

[1573] Input: Text data, emotion data

[1574] Output: Analysis results, saved in database

[1575] Step 6:

[1576] The server analyzes multiple opinions and performs summarization and topic extraction.

[1577] Specific operation: The server analyzes the received text data and summarizes important opinions and key phrases. It also extracts key topics based on the sentiment data. For example, the topic "expensive" and the sentiment "dissatisfied" are extracted.

[1578] Input: Text data, emotion data

[1579] Output: Summary data, topics

[1580] Step 7:

[1581] The server proposes improvement measures based on the summary data and emotion data.

[1582] Specific actions: The server uses the data it has learned in advance and past performance data to generate specific improvement measures from the analysis data. For example, it may suggest things like "run a sale" or "offer a special offer."

[1583] Input: Summary data, emotion data

[1584] Output: Improvements

[1585] Step 8:

[1586] The server compiles improvement measures into a report and sends it to the terminal.

[1587] Specific operation: The server compiles the proposed improvement measures into a report format and sends it to the terminal using the HTTP protocol. For example, an improvement measure such as "conducting a sale" is included in the report and sent to the terminal.

[1588] Input: Remedy

[1589] Output: Send report

[1590] Step 9:

[1591] The terminal receives the report and displays and notifies it through a user interface.

[1592] Specific operation: The device displays the received report on the user interface and notifies the user of the completion of the report through a notification function. For example, a notification saying "New improvement measures have been proposed" is displayed on the user's device.

[1593] Input: report

[1594] Output: report display, notification

[1595] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1596] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1597] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1598] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1599] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1600] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1601] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1602] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1603] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1604] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1605] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1606] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1607] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1608] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1609] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1610] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1611] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1612] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1613] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1614] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1615] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1616] The following is further disclosed regarding the above embodiment.

[1617] (Claim 1)

[1618] means for collecting user voice-input opinions;

[1619] A means for converting the collected voice into text data;

[1620] means for transmitting text data to a server;

[1621] means for analyzing and summarizing the text data received by the server;

[1622] A means for the server to propose improvement measures based on the analyzed data;

[1623] A system that includes a means for compiling the proposed improvements into a report and sending it to the terminal.

[1624] (Claim 2)

[1625] 10. The system of claim 1, further comprising means for extracting specific topics from the summarized data.

[1626] (Claim 3)

[1627] 10. The system of claim 1, further comprising means for notifying a user after generating a report based on the suggested remedial action.

[1628] "Example 1"

[1629] (Claim 1)

[1630] means for collecting user voice-input opinions;

[1631] A means for converting the collected voice into text data;

[1632] means for transmitting text data to a server;

[1633] means for analyzing and summarizing the text data received by the server;

[1634] A means for the server to propose improvement measures based on the analyzed data;

[1635] A means for compiling the proposed improvements into a report and transmitting it to the terminal;

[1636] The system includes a means for the terminal to display and notify reports through a user interface.

[1637] (Claim 2)

[1638] 10. The system of claim 1, further comprising means for the server to analyze the received text data using natural language processing algorithms to extract key phrases and major topics.

[1639] (Claim 3)

[1640] 10. The system of claim 1, further comprising means for the server to use the generative AI model to suggest improvements.

[1641] "Application Example 1"

[1642] (Claim 1)

[1643] means for collecting user voice-input opinions;

[1644] A means for converting the collected voice into text data;

[1645] means for transmitting text data to a server;

[1646] means for analyzing and summarizing the text data received by the server;

[1647] A means for the server to propose improvement measures based on the analyzed data;

[1648] A means for compiling the proposed improvements into a report and transmitting it to the terminal;

[1649] A system for content distribution services that includes means for collecting and analyzing users' voice opinions and proposing improvements to improve the quality of the service.

[1650] (Claim 2)

[1651] 10. The system of claim 1, further comprising: means for extracting specific topics from the summarized data and means for further extracting topics related to content distribution services.

[1652] (Claim 3)

[1653] 10. The system of claim 1, further comprising means for notifying the user and providing the remediation to the content creator after generating the report based on the remediation suggestions.

[1654] "Example 2: Combining Emotion Engines"

[1655] (Claim 1)

[1656] means for collecting user voice-input opinions;

[1657] A means for converting the collected voice into text data;

[1658] means for transmitting text data and emotion data to a server;

[1659] means for analyzing and summarizing the text data and emotion data received by the server;

[1660] A means for the server to propose improvement measures based on the analyzed text data and emotion data;

[1661] A system that includes a means for compiling the proposed improvements into a report and sending it to the terminal.

[1662] (Claim 2)

[1663] 10. The system of claim 1, further comprising means for extracting specific topics from the summarized data.

[1664] (Claim 3)

[1665] The system of claim 1, wherein the system suggests remedial measures based on the emotion data.

[1666] "Application example 2 when combining emotion engines"

[1667] (Claim 1)

[1668] means for collecting user voice-input opinions;

[1669] A means for converting the collected voice into text data;

[1670] means for transmitting text data and emotion data to a server;

[1671] means for analyzing and summarizing the text data and emotion data received by the server;

[1672] A means for the server to propose improvement measures based on the analyzed data and emotion data;

[1673] A means for compiling the proposed improvements into a report and transmitting it to the terminal;

[1674] means for collecting and analyzing emotion data from a user's voice input;

[1675] A means for collecting user voice opinions and presenting product improvement measures based on emotion data in a virtual store;

[1676] A system including:

[1677] (Claim 2)

[1678] 10. The system of claim 1, further comprising means for extracting specific topics from the summarized data.

[1679] (Claim 3)

[1680] 10. The system of claim 1, further comprising means for notifying a user after generating a report based on the suggested remedial action. [Explanation of symbols]

[1681] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting user voice-input opinions; A means for converting the collected voice into text data; means for transmitting text data to a server; means for analyzing and summarizing the text data received by the server; A means for the server to propose improvement measures based on the analyzed data; The system includes a means for compiling the proposed improvements into a report and sending it to the terminal.

2. The system of claim 1 further comprising means for extracting specific topics from the summarized data.

3. The system of claim 1 further comprising means for notifying a user after generating a report based on the suggested remedial action.

Images