System

Abstract

A system is provided.SOLUTION: This system includes a means for collecting the utterance of a user, a means for converting the collected utterance into text data, a means for performing natural language processing to the text data, and for extracting a keyword, a means for evaluating the usefulness of the text data based on the extracted keyword, and a means for notifying the user that the usefulness is evaluated to be high.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional knowledge sharing systems have had difficulty efficiently extracting and sharing useful information and knowledge from everyday conversations and comments made during work. This is because users are unaware of the value of their own comments and because that information is not properly documented. As a result, valuable knowledge remains buried, preventing improvements in the efficiency and knowledge base of the entire organization. [Means for solving the problem]

[0005] The present invention provides a system including means for collecting user comments and converting them into text data, means for applying natural language processing to the collected text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, and means for notifying users of information that has been evaluated as being highly useful. Furthermore, if the collected comments are voice data, they are converted into text data using voice recognition technology, and comments collected through communication tools are automatically collected. The system also includes means for providing additional information related to text data that has been evaluated as being highly useful, and the value of the information can be specifically evaluated by comparing it with a past database.

[0006] A "means" is a method or device designed to achieve a particular purpose.

[0007] "User" means any person or organization that uses the System.

[0008] "Utterances" refer to words spoken by a user verbally or through chat tools.

[0009] "Text data" refers to the result of converting voice data into text information.

[0010] "Natural language processing" is a general term for techniques and methods that allow computers to understand human language.

[0011] "Keywords" are words or phrases that have particular significance within a piece of text.

[0012] "Usefulness" refers to the degree to which information is useful to others.

[0013] "Notification" refers to the act of the system conveying a message or warning to the user.

[0014] "Additional information" refers to supplementary data or materials added to the original information.

[0015] A "database" is a system designed to store data in an organized manner and make it easy to search and query.

[0016] "Communication tools" are software and platforms that people use to exchange information with each other.

[0017] "Speech recognition technology" refers to the technology that converts voice into text data. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[0040] Overall system overview

[0041] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[0042] 1. Data Collection

[0043] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[0044] 2. Natural Language Processing (NLP)

[0045] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[0046] 3. Extraction of useful information

[0047] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0048] 4. Notices and Suggestions

[0049] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[0050] 5. Providing Additional Information

[0051] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0052] Specific examples

[0053] Suppose a user says during a video conference, "Please report on the progress of the new project." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing.

[0054] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[0055] In this way, the system according to the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[0059] Step 2:

[0060] The device monitors the user's speech in real time, sometimes using speech recognition technology to collect speech data, and sometimes collecting it directly as text data.

[0061] Step 3:

[0062] The device converts the collected voice data into text data using voice recognition technology, such as voice recognition software like the Google Speech-to-Text API.

[0063] Step 4:

[0064] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[0065] Step 5:

[0066] The server receives the text data sent from the terminal and temporarily stores it in a database.

[0067] Step 6:

[0068] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[0069] Step 7:

[0070] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[0071] Step 8:

[0072] The server notifies the user of information that has been evaluated as highly useful. The notification is made, for example, by sending a message to the user's device using the API of a chat tool.

[0073] Step 9:

[0074] The server suggests that useful information is contained in the notification message and that this information be documented and shared. The notification may also include specific instructions and suggested documentation formats.

[0075] Step 10:

[0076] The server provides additional relevant information as needed, such as detailed documentation of a specific technology or implementation methods based on past examples, for user reference.

[0077] In this way, the system effectively extracts useful insights from users' everyday comments and realizes the function of notifying and sharing them at the appropriate time.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] Conventional voice data collection and analysis systems lack the functionality to extract useful information from user speech in real time and appropriately notify and share it. This can result in important information being overlooked, making it difficult to contribute to the organization's overall knowledge base. Furthermore, there are issues with the accuracy of converting voice data into text data and the accuracy of analysis using natural language processing technology. There is a need for a system that can resolve these issues, quickly and accurately extract useful information from user speech, and share it within an organization.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for transmitting the text data to a central processing unit via a communication system, means for extracting keywords from the text data using natural language analysis technology, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user based on the evaluated usefulness, and means for providing related additional information. This makes it possible to quickly and accurately extract useful information from user comments and to appropriately notify and share it.

[0083] "Means for collecting user utterances" refers to devices or software for capturing voice and text data uttered by users.

[0084] The "means of converting collected speech into text data" refers to the process of converting speech data into text data using speech recognition technology or a conversion algorithm.

[0085] "Means for transmitting text data to a central processing unit via a communication system" refers to protocols or means for transmitting converted text data to a central processing unit such as a server or cloud using a communication network.

[0086] "Natural language analysis technology" is a technology for analyzing text data and understanding its content and context, and includes algorithms for extracting keywords and important phrases.

[0087] "Keyword extraction" is the process of using natural language processing techniques to identify important words and phrases from text data.

[0088] "Means for assessing the usefulness of text data" refers to the process of using machine learning models or algorithms to assess the value or importance of the text data based on extracted keywords.

[0089] "Means for notifying the user" refers to a method for communicating the evaluated information to the user, and includes email, chat tools, push notifications, etc.

[0090] The "means for providing additional relevant information" is a process of providing additional resources or documents that are useful to the user based on the evaluated text data.

[0091] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[0092] System hardware and software configuration

[0093] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[0094] Terminal

[0095] The device monitors the user's daily work and conversations in real time and converts them into text data using voice recognition technology. The device may be the user's personal computer or smartphone, and is equipped with a microphone and voice recognition software (e.g., Google Speech-to-Text or Microsoft Azure Speech Recognition) for highly accurate analysis of the voice data.

[0096] server

[0097] The server receives the text data sent from the device and analyzes it using natural language processing techniques (e.g., large-scale language models such as BERT or GPT-4). This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Furthermore, based on the extracted entity information, the server evaluates the usefulness of the text data using a machine learning model (e.g., a custom model using Scikit-learn or TensorFlow).

[0098] Notification means

[0099] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is realized, for example, using the API of the chat tool (e.g., Slack API or Microsoft Teams API). If necessary, additional related information (e.g., detailed documentation on a specific technology or implementation methods based on past examples) is also provided to the user.

[0100] Specific examples

[0101] Consider a scenario where a user says, "Please report on the progress of the new project" during a video conference. This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing technology.

[0102] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[0103] Prompt Sentence Examples

[0104] Below are some example prompts to input to a generative AI model:

[0105] > "Please collect what users say during video conferences in real time, convert it into text data, and send it to a server. Then, please explain in detail how the system will analyze the text data using natural language processing technology, extract useful information, and notify users. Please also include the names of any specific hardware or software."

[0106] As a result, the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[0107] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0108] Step 1: Data collection

[0109] The device monitors the user's speech in real time and collects audio data. The input is the user's voice, and once acquired, it is saved as audio data. Specifically, the device's microphone captures the audio and saves it as an audio file.

[0110] Step 2: Voice Recognition

[0111] The device converts the voice data collected in step 1 into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it calls the Google Speech-to-Text or Microsoft Azure Speech Recognition API to send voice data and receive text data.

[0112] Step 3: Send text data

[0113] The terminal sends the text data generated in step 2 to the server. The input is text data, and the output is text data sent over the network. Specifically, the text data is sent to the server using the HTTPS protocol.

[0114] Step 4: Receiving and saving text data

[0115] The server receives and stores the text data sent in step 3. The input is the text data received from the network, and the output is the text data stored in the server. Specifically, the server stores the received text data in a database.

[0116] Step 5: Natural Language Processing (NLP)

[0117] The server performs natural language analysis on the stored text data. The input is the text data, and the output is extracted keywords and phrases. Specifically, it uses the BERT model to analyze the text data and extract important entities.

[0118] Step 6: Usability evaluation

[0119] The server evaluates the usefulness of the text data based on the keywords and phrases extracted in step 5. The input is the keywords and phrases, and the output is a usefulness evaluation score. Specifically, it applies a machine learning model using Scikit-learn and calculates the usefulness score by referring to past data and frequency data.

[0120] Step 7: User Notification

[0121] The server notifies the user based on the text data that was rated as highly useful. The input is the usefulness rating score and the rated text data, and the output is a notification to the user. Specifically, it uses the API of the chat tool (e.g., Slack API) to send the rating result and a message to the user saying, "This information is important. We recommend that you document it and share it with your team."

[0122] Step 8: Provide additional information

[0123] The server provides the user with additional relevant information as needed. The input is the evaluated text data, and the output is additional information such as related documents and past cases. Specifically, the server searches the knowledge base within the server and attaches relevant materials when notifying the user.

[0124] Through these steps, the system can extract useful information from users' comments and quickly and accurately notify and share it.

[0125] (Application example 1)

[0126] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0127] In brick-and-mortar stores, it is difficult for staff to respond quickly and accurately to the diverse questions and requests of customers. Furthermore, because of the vast amount of information customers ask, it is unrealistic for staff to keep track of everything. As a result, the quality of customer service may decline, and customer satisfaction may decrease. The present invention aims to provide a system that solves these problems and improves customer service.

[0128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0129] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, and means for recognizing voice data to collect customer comments in a physical store and displaying related useful information on a smart gadget. This makes it possible to analyze customer comments in real time and provide useful information to staff immediately.

[0130] "User" refers to a person or organization that uses the system.

[0131] "Means for collecting speech" refers to equipment and software for collecting user voice data in real time.

[0132] "Means for converting into text data" refers to speech recognition technology or software for converting collected voice data into character string information.

[0133] "Natural language processing and keyword extraction methods" refers to algorithms and software used to identify and extract important words and phrases from text data.

[0134] "Means for assessing the usefulness of text data" refers to machine learning models and algorithms for assessing the value of information based on extracted keywords.

[0135] "Means of notification" refers to chat tools and applications for smart gadgets that inform users of information that has been rated as highly useful.

[0136] "Brick and mortar store" refers to a physical point of sale where goods or services are sold in person.

[0137] "Means for recognizing voice data" refers to voice recognition technology and devices used to collect and analyze customer speech within a physical store.

[0138] "Smart gadgets" refers to wearable or portable devices that can display information in real time, such as smart glasses or smartphones.

[0139] "Means for displaying relevant useful information" refers to a display and related software for visually providing useful information to the user.

[0140] Overall system overview

[0141] The system according to the present invention comprises a terminal that collects user comments in real time and transmits them as text data to a server, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and notification means that notifies the user of the useful information and provides additional information. A specific embodiment will be described below.

[0142] 1. Data Collection

[0143] The device monitors in real time what users and store staff say during their daily work and conversations, and converts this into text data using voice recognition technology. The device can be a smart eyeglass or a smartphone. A microphone and voice recognition software are built in to accurately analyze the voice data. For example, Microsoft Azure's voice recognition service is used.

[0144] 2. Natural Language Processing (NLP)

[0145] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as Google BERT and OpenAI GPT-3 to identify entities (important words and phrases) within the text.

[0146] 3. Extraction of useful information

[0147] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0148] 4. Notices and Suggestions

[0149] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[0150] 5. Providing Additional Information

[0151] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0152] Specific examples

[0153] For example, if a store staff member is asked by a customer, "Please tell me the stock status of this item," the system works as follows:

[0154] 1. Input prompt example

[0155] Customer: "What is the availability of this item?"

[0156] 2. Audio collection and recognition

[0157] Speech to text: "What is the availability of this item?"

[0158] 3. Natural Language Processing

[0159] Keyword extraction: "Stock status", "Product"

[0160] 4. Usefulness evaluation and notification

[0161] Check the registered product name and availability and provide specific answers such as "In stock" or "Out of stock" to staff

[0162] In this way, store staff can respond to customers in real time, improving the quality of service.The system of the present invention effectively extracts useful information from users' everyday utterances and appropriately notifies and shares it, thereby strengthening the knowledge base of the entire organization.

[0163] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0164] Step 1:

[0165] The device collects the audio data.

[0166] Input: User or customer speech (voice data)

[0167] How it works: The device's microphone collects the user's or customer's voice in real time.

[0168] Output: Collected audio data

[0169] Step 2:

[0170] The terminal converts the voice data into text data.

[0171] Input: Audio data

[0172] How it works: Speech recognition technology (for example, Microsoft Azure's speech recognition service) processes the voice data and converts it into text data.

[0173] Output: Text data

[0174] Step 3:

[0175] Sends text data to the server.

[0176] Input: Text data

[0177] Operation: The device sends the converted text data to a server via the Internet.

[0178] Output: Text data sent to the server

[0179] Step 4:

[0180] The server performs natural language processing on the received text data.

[0181] Input: Text data sent to the server

[0182] How it works: A server-based natural language processing engine (e.g., Google BERT or OpenAI GPT-3) analyzes text data and extracts important keywords and phrases.

[0183] Output: Extracted keywords and phrases

[0184] Step 5:

[0185] The server evaluates the usefulness of the text data based on the extracted keywords.

[0186] Input: Extracted keywords or phrases

[0187] How it works: The server's machine learning model uses historical data and other relevant information to determine the usefulness of the extracted keywords.

[0188] Output: Evaluation result (whether the information is useful or not)

[0189] Step 6:

[0190] The server notifies the user of the text data that has been evaluated as being highly useful.

[0191] Input: Evaluation results (highly useful information)

[0192] Operation: The server generates a notification message and sends it to the user through a notification means (for example, an API for a chat tool).

[0193] Output: Message notified to the user

[0194] Step 7:

[0195] The server will provide additional information as needed.

[0196] Input: Request and context information after user notification

[0197] How it works: The server searches for additional relevant information (e.g., technical documentation or past cases) and provides it to the user.

[0198] Output: Additional information displayed to the user

[0199] Specific examples

[0200] For example, if a customer asks, "Please tell me the stock status of this product," the voice data collected in steps 1-3 is converted into text data and sent to the server. In step 4, natural language processing is performed to extract keywords such as "stock status" and "product." In step 5, the usefulness is evaluated, and if it is determined to be useful, in step 6, the staff member is notified that "This product is in stock." Additional information such as the product's location and price is also provided in step 7.

[0201] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0202] The system according to the present invention collects user comments, evaluates the usefulness and emotions of the comments through natural language processing and an emotion engine, and notifies users appropriately. Specific embodiments for carrying out the present invention will be described below.

[0203] Overall system overview

[0204] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data to extract and evaluate useful information, an emotion engine that recognizes the user's emotions, and a means of notifying the user of the useful information and providing additional information.

[0205] 1. Data Collection

[0206] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[0207] 2. Natural Language Processing (NLP)

[0208] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[0209] 3. Extraction of useful information

[0210] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0211] 4. Emotion Engine

[0212] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[0213] 5. Notices and Suggestions

[0214] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[0215] 6. Providing Additional Information

[0216] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0217] Specific examples

[0218] Suppose a user says during a video conference, "This new project is very tough, but I'm working hard." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "tough" are extracted using natural language processing.

[0219] The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the user has the emotion "strict." If the evaluation result is deemed useful, the server notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." If the user is feeling stressed, this notification is further adjusted to include a message of encouragement and support. Additional information is also provided, including success stories of similar projects and documents on efficient project management methods.

[0220] In this way, the system of the present invention can strengthen the knowledge base of the entire organization by efficiently extracting useful insights from users' everyday comments and providing appropriate notifications and sharing that take emotions into consideration.

[0221] The processing flow will be explained below.

[0222] Step 1:

[0223] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[0224] Step 2:

[0225] The device monitors the user's speech in real time, and the monitored voice data is collected by the device's microphone.

[0226] Step 3:

[0227] The device converts the collected voice data into text data using voice recognition technology, such as the Google Speech-to-Text API.

[0228] Step 4:

[0229] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[0230] Step 5:

[0231] The server receives the text data sent from the terminal and temporarily stores it in a database.

[0232] Step 6:

[0233] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[0234] Step 7:

[0235] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[0236] Step 8:

[0237] The server passes the text data to an emotion engine to recognize the user's emotions. The emotion engine detects emotional tones and specific emotion keywords (e.g., joy, anger, sadness) from the text data.

[0238] Step 9:

[0239] The server takes emotional information into account when evaluating the usefulness of text data based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it adjusts the importance of the information.

[0240] Step 10:

[0241] The server notifies the user of information that is rated as highly useful. The notification is sent via the API of the chat tool the user is using.

[0242] Step 11:

[0243] The server adjusts the content and format of notifications based on the emotional assessment: for example, if the user is feeling stressed, notifications will be delivered in a tone of encouragement and support.

[0244] Step 12:

[0245] The server provides the user with additional relevant information as needed, such as detailed documentation of a particular technology or implementation instructions based on past experience.

[0246] Through the above processing steps, this system can effectively extract useful insights from users' everyday comments and strengthen the knowledge base of the entire organization by providing appropriate notifications and sharing that take emotions into consideration.

[0247] Example 2

[0248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0249] In conventional systems, the process of collecting user comments and evaluating useful information is fragmented, which leads to problems such as inadequate notification and provision of relevant additional information that takes user emotions into account. Furthermore, the accuracy of converting voice data into text is sometimes insufficient, resulting in issues of overall lack of efficiency and accuracy.

[0250] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for identifying the user's emotions from the text data, means for notifying the user when the usefulness is evaluated as high, and means for providing related additional information. This makes it possible to collect user comments in real time, evaluate their usefulness and emotions, and then provide appropriate notifications and information.

[0251] A "means for collecting user utterances" is a device or process that monitors users' voice and text utterances in real time and acquires them as data.

[0252] "Means for converting collected speech into text data" refers to a device or process that converts speech data into text format using natural language processing techniques.

[0253] A "means for performing natural language processing on text data to extract keywords" is a device or process that uses natural language processing techniques to identify important words and phrases within text.

[0254] The "means for evaluating the usefulness of text data based on extracted keywords" refers to a device or process that uses a machine learning model or algorithm to evaluate the usefulness of text based on extracted keywords.

[0255] A "means for identifying user emotions from text data" is a device or process that uses natural language processing technology to analyze the emotional tone and specific emotional keywords in text and identify the user's emotions.

[0256] The "means for notifying the user when the usefulness is evaluated as high" is a device or process that sends an alert or message to the user based on the text data that has been evaluated as being highly useful.

[0257] The "means for providing related additional information" is a device or process that provides a user with additional information, such as documents or examples, related to the evaluated text data.

[0258] The system of the present invention collects user utterances, evaluates the usefulness and sentiment of the utterances through natural language processing and an emotion engine, and provides appropriate notifications and related additional information.

[0259] Overall overview

[0260] The system consists of the following elements:

[0261] Device that collects comments

[0262] Software that converts collected comments into text data

[0263] A server that performs natural language processing on text data to extract and evaluate information

[0264] Emotion engine that recognizes user emotions

[0265] Means of notifying useful information

[0266] A means of providing additional relevant information

[0267] Data collection

[0268] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device uses common communication devices such as PCs and smartphones, and uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data with high accuracy.

[0269] Examples:

[0270] When a user says, "My new project is very challenging, but rewarding," the speech is collected through the device's microphone and converted into text data by speech recognition software.

[0271] Natural Language Processing (NLP)

[0272] The server receives the text data sent from the device and performs natural language processing using large-scale language models such as BERT to extract entities and important phrases within the text.

[0273] Examples:

[0274] The server analyzes the received text data, "The new project is very challenging, but rewarding," and extracts keywords such as "new project," "challenging," and "rewarding."

[0275] Extracting useful information

[0276] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords, and references past case studies and inquiry data stored in the company's database for the evaluation.

[0277] Examples:

[0278] If the keyword "new project" is information that has been evaluated as useful in the past, the current text data will also be evaluated as useful.

[0279] Emotion Engine

[0280] The emotion engine uses natural language processing technology to identify user emotions from text data, detecting emotions such as joy, challenge, and stress from keywords and context.

[0281] Examples:

[0282] From phrases such as "challenging," the user's emotion is identified as "positive challenge."

[0283] Notification of useful information

[0284] The server notifies the user of text data that has been evaluated as highly useful. This notification is made via a chat tool (e.g., Slack API).

[0285] Examples:

[0286] The user receives the message, "This information is important. We recommend that you document it and share it with your team."

[0287] Providing additional relevant information

[0288] The server will provide the user with additional relevant information as needed, including detailed documentation of past cases and technologies.

[0289] Examples:

[0290] Provide users with a link that says, "Here is detailed documentation of past success stories."

[0291] Prompt Sentence Examples

[0292] "The new project is very challenging. Please rate the usefulness and sentiment of this statement. Also, please provide any additional relevant information."

[0293] According to this invention, it is possible to collect and analyze user comments in real time and evaluate the usefulness and sentiment of the comments, thereby making it possible to provide appropriate notifications and information.

[0294] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0295] Step 1: Data collection

[0296] The device monitors the user's speech in real time and collects it as voice data. At this stage, the input is the user's speech into the microphone. The voice data is passed to the device's voice recognition software (e.g., Google Speech-to-Text API). The voice recognition software analyzes the voice data and converts it into text data. The output is the converted text data.

[0297] Specifically, the user says, "This new project is very challenging, but worthwhile." This is collected as voice data on the device.

[0298] Step 2: Sending text data

[0299] The device sends the text data converted from the voice data to the server. The input is the text data obtained in the previous step. The device sends this text data to the server using a RESTful API. The output is the text data received by the server.

[0300] Specifically, the text data "This new project is very challenging, but rewarding" is sent.

[0301] Step 3: Natural Language Processing (NLP)

[0302] The server analyzes the text data received from the device and extracts keywords and important phrases using natural language processing techniques. The input is the received text data. The server uses a large-scale language model such as BERT to identify important entities within the text. The output is the extracted keywords and phrases.

[0303] Specifically, the server extracts the keywords "new project," "challenging," and "rewarding" from the text "This new project is very challenging, but rewarding."

[0304] Step 4: Extract useful information

[0305] The server evaluates the usefulness of the text data based on the extracted keywords and phrases. The input is the extracted keywords. Using a machine learning model, the evaluation is performed by referencing past cases and inquiry data from an internal database. The output is a usefulness score for the text data.

[0306] As a concrete example, let's say the keyword "new project" has been frequently used in past highly rated cases. In this case, the text data will be evaluated as useful.

[0307] Step 5: Sentiment analysis

[0308] The server uses an emotion engine to analyze user emotions from text data. The input is text data. Natural language processing techniques are used to detect context and specific emotion keywords and identify emotions. The output is the detected emotion information.

[0309] As a specific operation, it identifies that the user has a positive feeling of challenge from the word "challenging."

[0310] Step 6: Notification

[0311] The server notifies the user based on the text data that was rated as highly useful and the results of sentiment analysis. The input is the usefulness score and sentiment information of the text data. The notification is sent using the chat tool's API (e.g., Slack API). The output is a notification message sent to the user.

[0312] Specifically, the system sends a message to the user saying, "This information is important. We recommend that you document it and share it with your team." Based on the sentiment, it also adds words of encouragement.

[0313] Step 7: Provide additional information

[0314] The server provides the user with additional related information as needed. The input is keywords in the text data and past case data. Related documents and case links are generated and provided. The output is the additional information provided to the user.

[0315] Specifically, the user is provided with a link that says, "Click here for detailed documentation on past success stories."

[0316] Through these steps, the system collects and analyzes user comments in real time, evaluates their usefulness and sentiment, and provides appropriate notifications and additional information.

[0317] (Application example 2)

[0318] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] In the past, customer service in brick-and-mortar stores did not analyze communication between customers and staff in real time, making it difficult to accurately grasp customer needs and emotions. Furthermore, there was a lack of a system for providing appropriate feedback and suggestions quickly, making it difficult to respond immediately to improve customer satisfaction. For this reason, there was a need for a way to improve staff response efficiency and increase customer satisfaction.

[0320] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0321] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, means for analyzing the user's emotions from the text data and adjusting the content of the notification based on the emotions, and means for providing related additional information under specific conditions. This makes it possible to analyze communication between customers and staff in real time and provide quick and appropriate feedback and suggestions based on the customer's needs and emotions.

[0322] "Means for collecting user utterances" refers to a voice input device for collecting voice conversations and utterances made by customers and staff in real time. Typically, this is done using a microphone built into smart glasses or a smartphone.

[0323] "Means for converting collected speech into text data" refers to technology that converts collected speech data into text data using speech recognition technology, including the Google Speech Recognition API and the SpeechRecognition library.

[0324] "Means of extracting keywords by applying natural language processing to text data" refers to natural language processing techniques that analyze text data and extract important words and phrases. Large-scale language models such as BERT and Transformers are used.

[0325] "Means for evaluating the usefulness of text data based on extracted keywords" refers to a machine learning model that analyzes extracted keywords and determines whether the text data is useful. Criteria used include internal historical data and entity frequency.

[0326] "Means for notifying users when information is rated as highly useful" refers to a means for notifying customers and staff of information that has been rated as useful. This can be done using the API of the chat application or a notification system.

[0327] "Means of analyzing user emotions from text data and adjusting notification content based on those emotions" refers to technology that analyzes the emotions of customers and staff from text data and appropriately adjusts the content and format of notifications based on the analysis results. Emotion analysis utilizes an emotion recognition pipeline.

[0328] "Means for providing relevant additional information under specific conditions" refers to means for providing relevant information that a user needs. This is a system that suggests additional information such as past cases, detailed documentation, and implementation methods.

[0329] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and an emotion engine that recognizes the user's emotions.It also includes a means to appropriately notify the user of useful information and provide additional information.

[0330] 1. Data Collection

[0331] The terminals, which are smart glasses or smartphones equipped with microphones and voice recognition software, monitor everyday conversations between customers and staff in real time and convert them into text using voice recognition technology.

[0332] 2. Natural Language Processing (NLP)

[0333] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This analysis involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as BERT to identify entities (important words and phrases) within the text.

[0334] 3. Extraction of useful information

[0335] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information. This evaluation is based on past data and the frequency of entities, and determines the usefulness of the information.

[0336] 4. Emotion Engine

[0337] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[0338] 5. Notices and Suggestions

[0339] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[0340] 6. Providing Additional Information

[0341] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0342] Specific examples

[0343] Suppose a customer says to a staff member wearing smart glasses, "This new product is a little difficult to use." This utterance is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new product" and "difficult to use" are extracted through natural language processing. The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the customer has negative emotions. If the evaluation result is deemed useful, the staff member is notified, saying, "It seems that the customer is dissatisfied with the new product. Please listen to their story in more detail and make suggestions to improve usability." An example of a prompt sentence is as follows:

[0344] Prompt Sentence Examples

[0345] "Japanese text data: 'This new product is a little difficult to use.' Please analyze it with a sentiment analysis tool."

[0346] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0347] Step 1:

[0348] The terminal monitors everyday conversations between customers and staff in real time and collects voice data. The collected voice data is captured using a microphone built into smart glasses or smartphones. The input is voice data, and the output is an audio file containing the recorded voice data.

[0349] Step 2:

[0350] The device converts the collected voice data into text data using voice recognition technology. The Google speech recognition API and SpeechRecognition library are used here. The input is voice data, and the output is text data converted from the voice data.

[0351] Step 3:

[0352] The server receives text data sent from the terminal. The input is text data, and the output is also text data. This step mainly involves data transfer.

[0353] Step 4:

[0354] The server performs natural language processing (NLP) on the received text data to extract keywords. Here, generative AI models such as BERT and Transformers are used to analyze sentences and identify important entities (words and phrases). The input is the text data, and the output is the extracted keywords.

[0355] Step 5:

[0356] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords. The evaluation uses criteria such as past data and entity frequency. The input is the extracted keywords, and the output is a score indicating the usefulness of the text data.

[0357] Step 6:

[0358] The server's emotion engine analyzes user emotions from text data. It uses an emotion recognition pipeline to detect emotional tones and specific emotion keywords (e.g., joy, anger, sadness) in the text. The input is text data, and the output is the analyzed emotion information.

[0359] Step 7:

[0360] The server notifies the user of text data that is rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is performed using the API of a chat tool. The input is the usefulness score and sentiment information of the text data, and the output is a notification message to the user.

[0361] Step 8:

[0362] The server adjusts the notification content and format based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will send a suggested message in a softer tone. This adjustment is made using the results of emotion analysis. The input is emotional information, and the output is the adjusted notification message.

[0363] Step 9:

[0364] The server provides users with additional information related to a specific condition, such as detailed documentation on a specific technology or implementation methods based on past examples. The input is text data evaluated as highly useful, and the output is the related additional information.

[0365] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0366] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0367] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0368] [Second embodiment]

[0369] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0370] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0371] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0372] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0373] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0374] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0375] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0376] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0377] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0378] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0379] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0380] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0381] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[0382] Overall system overview

[0383] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[0384] 1. Data Collection

[0385] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[0386] 2. Natural Language Processing (NLP)

[0387] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[0388] 3. Extraction of useful information

[0389] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0390] 4. Notices and Suggestions

[0391] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[0392] 5. Providing Additional Information

[0393] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0394] Specific examples

[0395] Suppose a user says during a video conference, "Please report on the progress of the new project." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing.

[0396] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[0397] In this way, the system according to the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[0398] The processing flow will be explained below.

[0399] Step 1:

[0400] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[0401] Step 2:

[0402] The device monitors the user's speech in real time, sometimes using speech recognition technology to collect speech data, and sometimes collecting it directly as text data.

[0403] Step 3:

[0404] The device converts the collected voice data into text data using voice recognition technology, such as voice recognition software like the Google Speech-to-Text API.

[0405] Step 4:

[0406] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[0407] Step 5:

[0408] The server receives the text data sent from the terminal and temporarily stores it in a database.

[0409] Step 6:

[0410] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[0411] Step 7:

[0412] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[0413] Step 8:

[0414] The server notifies the user of information that has been evaluated as highly useful. The notification is made, for example, by sending a message to the user's device using the API of a chat tool.

[0415] Step 9:

[0416] The server suggests that useful information is contained in the notification message and that this information be documented and shared. The notification may also include specific instructions and suggested documentation formats.

[0417] Step 10:

[0418] The server provides additional relevant information as needed, such as detailed documentation of a specific technology or implementation methods based on past examples, for user reference.

[0419] In this way, the system effectively extracts useful insights from users' everyday comments and realizes the function of notifying and sharing them at the appropriate time.

[0420] Example 1

[0421] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0422] Conventional voice data collection and analysis systems lack the functionality to extract useful information from user speech in real time and appropriately notify and share it. This can result in important information being overlooked, making it difficult to contribute to the organization's overall knowledge base. Furthermore, there are issues with the accuracy of converting voice data into text data and the accuracy of analysis using natural language processing technology. There is a need for a system that can resolve these issues, quickly and accurately extract useful information from user speech, and share it within an organization.

[0423] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0424] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for transmitting the text data to a central processing unit via a communication system, means for extracting keywords from the text data using natural language analysis technology, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user based on the evaluated usefulness, and means for providing related additional information. This makes it possible to quickly and accurately extract useful information from user comments and to appropriately notify and share it.

[0425] "Means for collecting user utterances" refers to devices or software for capturing voice and text data uttered by users.

[0426] The "means of converting collected speech into text data" refers to the process of converting speech data into text data using speech recognition technology or a conversion algorithm.

[0427] "Means for transmitting text data to a central processing unit via a communication system" refers to protocols or means for transmitting converted text data to a central processing unit such as a server or cloud using a communication network.

[0428] "Natural language analysis technology" is a technology for analyzing text data and understanding its content and context, and includes algorithms for extracting keywords and important phrases.

[0429] "Keyword extraction" is the process of using natural language processing techniques to identify important words and phrases from text data.

[0430] "Means for assessing the usefulness of text data" refers to the process of using machine learning models or algorithms to assess the value or importance of the text data based on extracted keywords.

[0431] "Means for notifying the user" refers to a method for communicating the evaluated information to the user, and includes email, chat tools, push notifications, etc.

[0432] The "means for providing additional relevant information" is a process of providing additional resources or documents that are useful to the user based on the evaluated text data.

[0433] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[0434] System hardware and software configuration

[0435] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[0436] Terminal

[0437] The device monitors the user's daily work and conversations in real time and converts them into text data using voice recognition technology. The device may be the user's personal computer or smartphone, and is equipped with a microphone and voice recognition software (e.g., Google Speech-to-Text or Microsoft Azure Speech Recognition) for highly accurate analysis of the voice data.

[0438] server

[0439] The server receives the text data sent from the device and analyzes it using natural language processing techniques (e.g., large-scale language models such as BERT or GPT-4). This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Furthermore, based on the extracted entity information, the server evaluates the usefulness of the text data using a machine learning model (e.g., a custom model using Scikit-learn or TensorFlow).

[0440] Notification means

[0441] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is realized, for example, using the API of the chat tool (e.g., Slack API or Microsoft Teams API). If necessary, additional related information (e.g., detailed documentation on a specific technology or implementation methods based on past examples) is also provided to the user.

[0442] Specific examples

[0443] Consider a scenario where a user says, "Please report on the progress of the new project" during a video conference. This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing technology.

[0444] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[0445] Prompt Sentence Examples

[0446] Below are some example prompts to input to a generative AI model:

[0447] > "Please collect what users say during video conferences in real time, convert it into text data, and send it to a server. Then, please explain in detail how the system will analyze the text data using natural language processing technology, extract useful information, and notify users. Please also include the names of any specific hardware or software."

[0448] As a result, the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[0449] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0450] Step 1: Data collection

[0451] The device monitors the user's speech in real time and collects audio data. The input is the user's voice, and once acquired, it is saved as audio data. Specifically, the device's microphone captures the audio and saves it as an audio file.

[0452] Step 2: Voice Recognition

[0453] The device converts the voice data collected in step 1 into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it calls the Google Speech-to-Text or Microsoft Azure Speech Recognition API to send voice data and receive text data.

[0454] Step 3: Send text data

[0455] The terminal sends the text data generated in step 2 to the server. The input is text data, and the output is text data sent over the network. Specifically, the text data is sent to the server using the HTTPS protocol.

[0456] Step 4: Receiving and saving text data

[0457] The server receives and stores the text data sent in step 3. The input is the text data received from the network, and the output is the text data stored in the server. Specifically, the server stores the received text data in a database.

[0458] Step 5: Natural Language Processing (NLP)

[0459] The server performs natural language analysis on the stored text data. The input is the text data, and the output is extracted keywords and phrases. Specifically, it uses the BERT model to analyze the text data and extract important entities.

[0460] Step 6: Usability evaluation

[0461] The server evaluates the usefulness of the text data based on the keywords and phrases extracted in step 5. The input is the keywords and phrases, and the output is a usefulness evaluation score. Specifically, it applies a machine learning model using Scikit-learn and calculates the usefulness score by referring to past data and frequency data.

[0462] Step 7: User Notification

[0463] The server notifies the user based on the text data that was rated as highly useful. The input is the usefulness rating score and the rated text data, and the output is a notification to the user. Specifically, it uses the API of the chat tool (e.g., Slack API) to send the rating result and a message to the user saying, "This information is important. We recommend that you document it and share it with your team."

[0464] Step 8: Provide additional information

[0465] The server provides the user with additional relevant information as needed. The input is the evaluated text data, and the output is additional information such as related documents and past cases. Specifically, the server searches the knowledge base within the server and attaches relevant materials when notifying the user.

[0466] Through these steps, the system can extract useful information from users' comments and quickly and accurately notify and share it.

[0467] (Application example 1)

[0468] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0469] In brick-and-mortar stores, it is difficult for staff to respond quickly and accurately to the diverse questions and requests of customers. Furthermore, because of the vast amount of information customers ask, it is unrealistic for staff to keep track of everything. As a result, the quality of customer service may decline, and customer satisfaction may decrease. The present invention aims to provide a system that solves these problems and improves customer service.

[0470] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0471] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, and means for recognizing voice data to collect customer comments in a physical store and displaying related useful information on a smart gadget. This makes it possible to analyze customer comments in real time and provide useful information to staff immediately.

[0472] "User" refers to a person or organization that uses the system.

[0473] "Means for collecting speech" refers to equipment and software for collecting user voice data in real time.

[0474] "Means for converting into text data" refers to speech recognition technology or software for converting collected voice data into character string information.

[0475] "Natural language processing and keyword extraction methods" refers to algorithms and software used to identify and extract important words and phrases from text data.

[0476] "Means for assessing the usefulness of text data" refers to machine learning models and algorithms for assessing the value of information based on extracted keywords.

[0477] "Means of notification" refers to chat tools and applications for smart gadgets that inform users of information that has been rated as highly useful.

[0478] "Brick and mortar store" refers to a physical point of sale where goods or services are sold in person.

[0479] "Means for recognizing voice data" refers to voice recognition technology and devices used to collect and analyze customer speech within a physical store.

[0480] "Smart gadgets" refers to wearable or portable devices that can display information in real time, such as smart glasses or smartphones.

[0481] "Means for displaying relevant useful information" refers to a display and related software for visually providing useful information to the user.

[0482] Overall system overview

[0483] The system according to the present invention comprises a terminal that collects user comments in real time and transmits them as text data to a server, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and notification means that notifies the user of the useful information and provides additional information. A specific embodiment will be described below.

[0484] 1. Data Collection

[0485] The device monitors in real time what users and store staff say during their daily work and conversations, and converts this into text data using voice recognition technology. The device can be a smart eyeglass or a smartphone. A microphone and voice recognition software are built in to accurately analyze the voice data. For example, Microsoft Azure's voice recognition service is used.

[0486] 2. Natural Language Processing (NLP)

[0487] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as Google BERT and OpenAI GPT-3 to identify entities (important words and phrases) within the text.

[0488] 3. Extraction of useful information

[0489] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0490] 4. Notices and Suggestions

[0491] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[0492] 5. Providing Additional Information

[0493] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0494] Specific examples

[0495] For example, if a store staff member is asked by a customer, "Please tell me the stock status of this item," the system works as follows:

[0496] 1. Input prompt example

[0497] Customer: "What is the availability of this item?"

[0498] 2. Audio collection and recognition

[0499] Speech to text: "What is the availability of this item?"

[0500] 3. Natural Language Processing

[0501] Keyword extraction: "Stock status", "Product"

[0502] 4. Usefulness evaluation and notification

[0503] Check the registered product name and availability and provide specific answers such as "In stock" or "Out of stock" to staff

[0504] In this way, store staff can respond to customers in real time, improving the quality of service.The system of the present invention effectively extracts useful information from users' everyday utterances and appropriately notifies and shares it, thereby strengthening the knowledge base of the entire organization.

[0505] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0506] Step 1:

[0507] The device collects the audio data.

[0508] Input: User or customer speech (voice data)

[0509] How it works: The device's microphone collects the user's or customer's voice in real time.

[0510] Output: Collected audio data

[0511] Step 2:

[0512] The terminal converts the voice data into text data.

[0513] Input: Audio data

[0514] How it works: Speech recognition technology (for example, Microsoft Azure's speech recognition service) processes the voice data and converts it into text data.

[0515] Output: Text data

[0516] Step 3:

[0517] Sends text data to the server.

[0518] Input: Text data

[0519] Operation: The device sends the converted text data to a server via the Internet.

[0520] Output: Text data sent to the server

[0521] Step 4:

[0522] The server performs natural language processing on the received text data.

[0523] Input: Text data sent to the server

[0524] How it works: A server-based natural language processing engine (e.g., Google BERT or OpenAI GPT-3) analyzes text data and extracts important keywords and phrases.

[0525] Output: Extracted keywords and phrases

[0526] Step 5:

[0527] The server evaluates the usefulness of the text data based on the extracted keywords.

[0528] Input: Extracted keywords or phrases

[0529] How it works: The server's machine learning model uses historical data and other relevant information to determine the usefulness of the extracted keywords.

[0530] Output: Evaluation result (whether the information is useful or not)

[0531] Step 6:

[0532] The server notifies the user of the text data that has been evaluated as being highly useful.

[0533] Input: Evaluation results (highly useful information)

[0534] Operation: The server generates a notification message and sends it to the user through a notification means (for example, an API for a chat tool).

[0535] Output: Message notified to the user

[0536] Step 7:

[0537] The server will provide additional information as needed.

[0538] Input: Request and context information after user notification

[0539] How it works: The server searches for additional relevant information (e.g., technical documentation or past cases) and provides it to the user.

[0540] Output: Additional information displayed to the user

[0541] Specific examples

[0542] For example, if a customer asks, "Please tell me the stock status of this product," the voice data collected in steps 1-3 is converted into text data and sent to the server. In step 4, natural language processing is performed to extract keywords such as "stock status" and "product." In step 5, the usefulness is evaluated, and if it is determined to be useful, in step 6, the staff member is notified that "This product is in stock." Additional information such as the product's location and price is also provided in step 7.

[0543] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0544] The system according to the present invention collects user comments, evaluates the usefulness and emotions of the comments through natural language processing and an emotion engine, and notifies users appropriately. Specific embodiments for carrying out the present invention will be described below.

[0545] Overall system overview

[0546] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data to extract and evaluate useful information, an emotion engine that recognizes the user's emotions, and a means of notifying the user of the useful information and providing additional information.

[0547] 1. Data Collection

[0548] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[0549] 2. Natural Language Processing (NLP)

[0550] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[0551] 3. Extraction of useful information

[0552] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0553] 4. Emotion Engine

[0554] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[0555] 5. Notices and Suggestions

[0556] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[0557] 6. Providing Additional Information

[0558] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0559] Specific examples

[0560] Suppose a user says during a video conference, "This new project is very tough, but I'm working hard." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "tough" are extracted using natural language processing.

[0561] The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the user has the emotion "strict." If the evaluation result is deemed useful, the server notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." If the user is feeling stressed, this notification is further adjusted to include a message of encouragement and support. Additional information is also provided, including success stories of similar projects and documents on efficient project management methods.

[0562] In this way, the system of the present invention can strengthen the knowledge base of the entire organization by efficiently extracting useful insights from users' everyday comments and providing appropriate notifications and sharing that take emotions into consideration.

[0563] The processing flow will be explained below.

[0564] Step 1:

[0565] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[0566] Step 2:

[0567] The device monitors the user's speech in real time, and the monitored voice data is collected by the device's microphone.

[0568] Step 3:

[0569] The device converts the collected voice data into text data using voice recognition technology, such as the Google Speech-to-Text API.

[0570] Step 4:

[0571] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[0572] Step 5:

[0573] The server receives the text data sent from the terminal and temporarily stores it in a database.

[0574] Step 6:

[0575] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[0576] Step 7:

[0577] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[0578] Step 8:

[0579] The server passes the text data to an emotion engine to recognize the user's emotions. The emotion engine detects emotional tones and specific emotion keywords (e.g., joy, anger, sadness) from the text data.

[0580] Step 9:

[0581] The server takes emotional information into account when evaluating the usefulness of text data based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it adjusts the importance of the information.

[0582] Step 10:

[0583] The server notifies the user of information that is rated as highly useful. The notification is sent via the API of the chat tool the user is using.

[0584] Step 11:

[0585] The server adjusts the content and format of notifications based on the emotional assessment: for example, if the user is feeling stressed, notifications will be delivered in a tone of encouragement and support.

[0586] Step 12:

[0587] The server provides the user with additional relevant information as needed, such as detailed documentation of a particular technology or implementation instructions based on past experience.

[0588] Through the above processing steps, this system can effectively extract useful insights from users' everyday comments and strengthen the knowledge base of the entire organization by providing appropriate notifications and sharing that take emotions into consideration.

[0589] Example 2

[0590] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0591] In conventional systems, the process of collecting user comments and evaluating useful information is fragmented, which leads to problems such as inadequate notification and provision of relevant additional information that takes user emotions into account. Furthermore, the accuracy of converting voice data into text is sometimes insufficient, resulting in issues of overall lack of efficiency and accuracy.

[0592] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for identifying the user's emotions from the text data, means for notifying the user when the usefulness is evaluated as high, and means for providing related additional information. This makes it possible to collect user comments in real time, evaluate their usefulness and emotions, and then provide appropriate notifications and information.

[0593] A "means for collecting user utterances" is a device or process that monitors users' voice and text utterances in real time and acquires them as data.

[0594] "Means for converting collected speech into text data" refers to a device or process that converts speech data into text format using natural language processing techniques.

[0595] A "means for performing natural language processing on text data to extract keywords" is a device or process that uses natural language processing techniques to identify important words and phrases within text.

[0596] The "means for evaluating the usefulness of text data based on extracted keywords" refers to a device or process that uses a machine learning model or algorithm to evaluate the usefulness of text based on extracted keywords.

[0597] A "means for identifying user emotions from text data" is a device or process that uses natural language processing technology to analyze the emotional tone and specific emotional keywords in text and identify the user's emotions.

[0598] The "means for notifying the user when the usefulness is evaluated as high" is a device or process that sends an alert or message to the user based on the text data that has been evaluated as being highly useful.

[0599] The "means for providing related additional information" is a device or process that provides a user with additional information, such as documents or examples, related to the evaluated text data.

[0600] The system of the present invention collects user utterances, evaluates the usefulness and sentiment of the utterances through natural language processing and an emotion engine, and provides appropriate notifications and related additional information.

[0601] Overall overview

[0602] The system consists of the following elements:

[0603] Device that collects comments

[0604] Software that converts collected comments into text data

[0605] A server that performs natural language processing on text data to extract and evaluate information

[0606] Emotion engine that recognizes user emotions

[0607] Means of notifying useful information

[0608] A means of providing additional relevant information

[0609] Data collection

[0610] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device uses common communication devices such as PCs and smartphones, and uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data with high accuracy.

[0611] Examples:

[0612] When a user says, "My new project is very challenging, but rewarding," the speech is collected through the device's microphone and converted into text data by speech recognition software.

[0613] Natural Language Processing (NLP)

[0614] The server receives the text data sent from the device and performs natural language processing using large-scale language models such as BERT to extract entities and important phrases within the text.

[0615] Examples:

[0616] The server analyzes the received text data, "The new project is very challenging, but rewarding," and extracts keywords such as "new project," "challenging," and "rewarding."

[0617] Extracting useful information

[0618] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords, and references past case studies and inquiry data stored in the company's database for the evaluation.

[0619] Examples:

[0620] If the keyword "new project" is information that has been evaluated as useful in the past, the current text data will also be evaluated as useful.

[0621] Emotion Engine

[0622] The emotion engine uses natural language processing technology to identify user emotions from text data, detecting emotions such as joy, challenge, and stress from keywords and context.

[0623] Examples:

[0624] From phrases such as "challenging," the user's emotion is identified as "positive challenge."

[0625] Notification of useful information

[0626] The server notifies the user of text data that has been evaluated as highly useful. This notification is made via a chat tool (e.g., Slack API).

[0627] Examples:

[0628] The user receives the message, "This information is important. We recommend that you document it and share it with your team."

[0629] Providing additional relevant information

[0630] The server will provide the user with additional relevant information as needed, including detailed documentation of past cases and technologies.

[0631] Examples:

[0632] Provide users with a link that says, "Here is detailed documentation of past success stories."

[0633] Prompt Sentence Examples

[0634] "The new project is very challenging. Please rate the usefulness and sentiment of this statement. Also, please provide any additional relevant information."

[0635] According to this invention, it is possible to collect and analyze user comments in real time and evaluate the usefulness and sentiment of the comments, thereby making it possible to provide appropriate notifications and information.

[0636] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0637] Step 1: Data collection

[0638] The device monitors the user's speech in real time and collects it as voice data. At this stage, the input is the user's speech into the microphone. The voice data is passed to the device's voice recognition software (e.g., Google Speech-to-Text API). The voice recognition software analyzes the voice data and converts it into text data. The output is the converted text data.

[0639] Specifically, the user says, "This new project is very challenging, but worthwhile." This is collected as voice data on the device.

[0640] Step 2: Sending text data

[0641] The device sends the text data converted from the voice data to the server. The input is the text data obtained in the previous step. The device sends this text data to the server using a RESTful API. The output is the text data received by the server.

[0642] Specifically, the text data "This new project is very challenging, but rewarding" is sent.

[0643] Step 3: Natural Language Processing (NLP)

[0644] The server analyzes the text data received from the device and extracts keywords and important phrases using natural language processing techniques. The input is the received text data. The server uses a large-scale language model such as BERT to identify important entities within the text. The output is the extracted keywords and phrases.

[0645] Specifically, the server extracts the keywords "new project," "challenging," and "rewarding" from the text "This new project is very challenging, but rewarding."

[0646] Step 4: Extract useful information

[0647] The server evaluates the usefulness of the text data based on the extracted keywords and phrases. The input is the extracted keywords. Using a machine learning model, the evaluation is performed by referencing past cases and inquiry data from an internal database. The output is a usefulness score for the text data.

[0648] As a concrete example, let's say the keyword "new project" has been frequently used in past highly rated cases. In this case, the text data will be evaluated as useful.

[0649] Step 5: Sentiment analysis

[0650] The server uses an emotion engine to analyze user emotions from text data. The input is text data. Natural language processing techniques are used to detect context and specific emotion keywords and identify emotions. The output is the detected emotion information.

[0651] As a specific operation, it identifies that the user has a positive feeling of challenge from the word "challenging."

[0652] Step 6: Notification

[0653] The server notifies the user based on the text data that was rated as highly useful and the results of sentiment analysis. The input is the usefulness score and sentiment information of the text data. The notification is sent using the chat tool's API (e.g., Slack API). The output is a notification message sent to the user.

[0654] Specifically, the system sends a message to the user saying, "This information is important. We recommend that you document it and share it with your team." Based on the sentiment, it also adds words of encouragement.

[0655] Step 7: Provide additional information

[0656] The server provides the user with additional related information as needed. The input is keywords in the text data and past case data. Related documents and case links are generated and provided. The output is the additional information provided to the user.

[0657] Specifically, the user is provided with a link that says, "Click here for detailed documentation on past success stories."

[0658] Through these steps, the system collects and analyzes user comments in real time, evaluates their usefulness and sentiment, and provides appropriate notifications and additional information.

[0659] (Application example 2)

[0660] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0661] In the past, customer service in brick-and-mortar stores did not analyze communication between customers and staff in real time, making it difficult to accurately grasp customer needs and emotions. Furthermore, there was a lack of a system for providing appropriate feedback and suggestions quickly, making it difficult to respond immediately to improve customer satisfaction. For this reason, there was a need for a way to improve staff response efficiency and increase customer satisfaction.

[0662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0663] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, means for analyzing the user's emotions from the text data and adjusting the content of the notification based on the emotions, and means for providing related additional information under specific conditions. This makes it possible to analyze communication between customers and staff in real time and provide quick and appropriate feedback and suggestions based on the customer's needs and emotions.

[0664] "Means for collecting user utterances" refers to a voice input device for collecting voice conversations and utterances made by customers and staff in real time. Typically, this is done using a microphone built into smart glasses or a smartphone.

[0665] "Means for converting collected speech into text data" refers to technology that converts collected speech data into text data using speech recognition technology, including the Google Speech Recognition API and the SpeechRecognition library.

[0666] "Means of extracting keywords by applying natural language processing to text data" refers to natural language processing techniques that analyze text data and extract important words and phrases. Large-scale language models such as BERT and Transformers are used.

[0667] "Means for evaluating the usefulness of text data based on extracted keywords" refers to a machine learning model that analyzes extracted keywords and determines whether the text data is useful. Criteria used include internal historical data and entity frequency.

[0668] "Means for notifying users when information is rated as highly useful" refers to a means for notifying customers and staff of information that has been rated as useful. This can be done using the API of the chat application or a notification system.

[0669] "Means of analyzing user emotions from text data and adjusting notification content based on those emotions" refers to technology that analyzes the emotions of customers and staff from text data and appropriately adjusts the content and format of notifications based on the analysis results. Emotion analysis utilizes an emotion recognition pipeline.

[0670] "Means for providing relevant additional information under specific conditions" refers to means for providing relevant information that a user needs. This is a system that suggests additional information such as past cases, detailed documentation, and implementation methods.

[0671] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and an emotion engine that recognizes the user's emotions.It also includes a means to appropriately notify the user of useful information and provide additional information.

[0672] 1. Data Collection

[0673] The terminals, which are smart glasses or smartphones equipped with microphones and voice recognition software, monitor everyday conversations between customers and staff in real time and convert them into text using voice recognition technology.

[0674] 2. Natural Language Processing (NLP)

[0675] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This analysis involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as BERT to identify entities (important words and phrases) within the text.

[0676] 3. Extraction of useful information

[0677] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information. This evaluation is based on past data and the frequency of entities, and determines the usefulness of the information.

[0678] 4. Emotion Engine

[0679] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[0680] 5. Notices and Suggestions

[0681] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[0682] 6. Providing Additional Information

[0683] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0684] Specific examples

[0685] Suppose a customer says to a staff member wearing smart glasses, "This new product is a little difficult to use." This utterance is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new product" and "difficult to use" are extracted through natural language processing. The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the customer has negative emotions. If the evaluation result is deemed useful, the staff member is notified, saying, "It seems that the customer is dissatisfied with the new product. Please listen to their story in more detail and make suggestions to improve usability." An example of a prompt sentence is as follows:

[0686] Prompt Sentence Examples

[0687] "Japanese text data: 'This new product is a little difficult to use.' Please analyze it with a sentiment analysis tool."

[0688] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0689] Step 1:

[0690] The terminal monitors everyday conversations between customers and staff in real time and collects voice data. The collected voice data is captured using a microphone built into smart glasses or smartphones. The input is voice data, and the output is an audio file containing the recorded voice data.

[0691] Step 2:

[0692] The device converts the collected voice data into text data using voice recognition technology. The Google speech recognition API and SpeechRecognition library are used here. The input is voice data, and the output is text data converted from the voice data.

[0693] Step 3:

[0694] The server receives text data sent from the terminal. The input is text data, and the output is also text data. This step mainly involves data transfer.

[0695] Step 4:

[0696] The server performs natural language processing (NLP) on the received text data to extract keywords. Here, generative AI models such as BERT and Transformers are used to analyze sentences and identify important entities (words and phrases). The input is the text data, and the output is the extracted keywords.

[0697] Step 5:

[0698] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords. The evaluation uses criteria such as past data and entity frequency. The input is the extracted keywords, and the output is a score indicating the usefulness of the text data.

[0699] Step 6:

[0700] The server's emotion engine analyzes user emotions from text data. It uses an emotion recognition pipeline to detect emotional tones and specific emotion keywords (e.g., joy, anger, sadness) in the text. The input is text data, and the output is the analyzed emotion information.

[0701] Step 7:

[0702] The server notifies the user of text data that is rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is performed using the API of a chat tool. The input is the usefulness score and sentiment information of the text data, and the output is a notification message to the user.

[0703] Step 8:

[0704] The server adjusts the notification content and format based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will send a suggested message in a softer tone. This adjustment is made using the results of emotion analysis. The input is emotional information, and the output is the adjusted notification message.

[0705] Step 9:

[0706] The server provides users with additional information related to a specific condition, such as detailed documentation on a specific technology or implementation methods based on past examples. The input is text data evaluated as highly useful, and the output is the related additional information.

[0707] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0708] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0709] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0710] [Third embodiment]

[0711] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0712] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0713] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0714] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0715] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0716] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0717] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0718] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0719] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0720] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0721] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0722] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0723] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[0724] Overall system overview

[0725] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[0726] 1. Data Collection

[0727] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[0728] 2. Natural Language Processing (NLP)

[0729] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[0730] 3. Extraction of useful information

[0731] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0732] 4. Notices and Suggestions

[0733] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[0734] 5. Providing Additional Information

[0735] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0736] Specific examples

[0737] Suppose a user says during a video conference, "Please report on the progress of the new project." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing.

[0738] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[0739] In this way, the system according to the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[0740] The processing flow will be explained below.

[0741] Step 1:

[0742] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[0743] Step 2:

[0744] The device monitors the user's speech in real time, sometimes using speech recognition technology to collect speech data, and sometimes collecting it directly as text data.

[0745] Step 3:

[0746] The device converts the collected voice data into text data using voice recognition technology, such as voice recognition software like the Google Speech-to-Text API.

[0747] Step 4:

[0748] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[0749] Step 5:

[0750] The server receives the text data sent from the terminal and temporarily stores it in a database.

[0751] Step 6:

[0752] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[0753] Step 7:

[0754] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[0755] Step 8:

[0756] The server notifies the user of information that has been evaluated as highly useful. The notification is made, for example, by sending a message to the user's device using the API of a chat tool.

[0757] Step 9:

[0758] The server suggests that useful information is contained in the notification message and that this information be documented and shared. The notification may also include specific instructions and suggested documentation formats.

[0759] Step 10:

[0760] The server provides additional relevant information as needed, such as detailed documentation of a specific technology or implementation methods based on past examples, for user reference.

[0761] In this way, the system effectively extracts useful insights from users' everyday comments and realizes the function of notifying and sharing them at the appropriate time.

[0762] Example 1

[0763] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0764] Conventional voice data collection and analysis systems lack the functionality to extract useful information from user speech in real time and appropriately notify and share it. This can result in important information being overlooked, making it difficult to contribute to the organization's overall knowledge base. Furthermore, there are issues with the accuracy of converting voice data into text data and the accuracy of analysis using natural language processing technology. There is a need for a system that can resolve these issues, quickly and accurately extract useful information from user speech, and share it within an organization.

[0765] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0766] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for transmitting the text data to a central processing unit via a communication system, means for extracting keywords from the text data using natural language analysis technology, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user based on the evaluated usefulness, and means for providing related additional information. This makes it possible to quickly and accurately extract useful information from user comments and to appropriately notify and share it.

[0767] "Means for collecting user utterances" refers to devices or software for capturing voice and text data uttered by users.

[0768] The "means of converting collected speech into text data" refers to the process of converting speech data into text data using speech recognition technology or a conversion algorithm.

[0769] "Means for transmitting text data to a central processing unit via a communication system" refers to protocols or means for transmitting converted text data to a central processing unit such as a server or cloud using a communication network.

[0770] "Natural language analysis technology" is a technology for analyzing text data and understanding its content and context, and includes algorithms for extracting keywords and important phrases.

[0771] "Keyword extraction" is the process of using natural language processing techniques to identify important words and phrases from text data.

[0772] "Means for assessing the usefulness of text data" refers to the process of using machine learning models or algorithms to assess the value or importance of the text data based on extracted keywords.

[0773] "Means for notifying the user" refers to a method for communicating the evaluated information to the user, and includes email, chat tools, push notifications, etc.

[0774] The "means for providing additional relevant information" is a process of providing additional resources or documents that are useful to the user based on the evaluated text data.

[0775] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[0776] System hardware and software configuration

[0777] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[0778] Terminal

[0779] The device monitors the user's daily work and conversations in real time and converts them into text data using voice recognition technology. The device may be the user's personal computer or smartphone, and is equipped with a microphone and voice recognition software (e.g., Google Speech-to-Text or Microsoft Azure Speech Recognition) for highly accurate analysis of the voice data.

[0780] server

[0781] The server receives the text data sent from the device and analyzes it using natural language processing techniques (e.g., large-scale language models such as BERT or GPT-4). This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Furthermore, based on the extracted entity information, the server evaluates the usefulness of the text data using a machine learning model (e.g., a custom model using Scikit-learn or TensorFlow).

[0782] Notification means

[0783] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is realized, for example, using the API of the chat tool (e.g., Slack API or Microsoft Teams API). If necessary, additional related information (e.g., detailed documentation on a specific technology or implementation methods based on past examples) is also provided to the user.

[0784] Specific examples

[0785] Consider a scenario where a user says, "Please report on the progress of the new project" during a video conference. This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing technology.

[0786] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[0787] Prompt Sentence Examples

[0788] Below are some example prompts to input to a generative AI model:

[0789] > "Please collect what users say during video conferences in real time, convert it into text data, and send it to a server. Then, please explain in detail how the system will analyze the text data using natural language processing technology, extract useful information, and notify users. Please also include the names of any specific hardware or software."

[0790] As a result, the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[0791] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0792] Step 1: Data collection

[0793] The device monitors the user's speech in real time and collects audio data. The input is the user's voice, and once acquired, it is saved as audio data. Specifically, the device's microphone captures the audio and saves it as an audio file.

[0794] Step 2: Voice Recognition

[0795] The device converts the voice data collected in step 1 into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it calls the Google Speech-to-Text or Microsoft Azure Speech Recognition API to send voice data and receive text data.

[0796] Step 3: Send text data

[0797] The terminal sends the text data generated in step 2 to the server. The input is text data, and the output is text data sent over the network. Specifically, the text data is sent to the server using the HTTPS protocol.

[0798] Step 4: Receiving and saving text data

[0799] The server receives and stores the text data sent in step 3. The input is the text data received from the network, and the output is the text data stored in the server. Specifically, the server stores the received text data in a database.

[0800] Step 5: Natural Language Processing (NLP)

[0801] The server performs natural language analysis on the stored text data. The input is the text data, and the output is extracted keywords and phrases. Specifically, it uses the BERT model to analyze the text data and extract important entities.

[0802] Step 6: Usability evaluation

[0803] The server evaluates the usefulness of the text data based on the keywords and phrases extracted in step 5. The input is the keywords and phrases, and the output is a usefulness evaluation score. Specifically, it applies a machine learning model using Scikit-learn and calculates the usefulness score by referring to past data and frequency data.

[0804] Step 7: User Notification

[0805] The server notifies the user based on the text data that was rated as highly useful. The input is the usefulness rating score and the rated text data, and the output is a notification to the user. Specifically, it uses the API of the chat tool (e.g., Slack API) to send the rating result and a message to the user saying, "This information is important. We recommend that you document it and share it with your team."

[0806] Step 8: Provide additional information

[0807] The server provides the user with additional relevant information as needed. The input is the evaluated text data, and the output is additional information such as related documents and past cases. Specifically, the server searches the knowledge base within the server and attaches relevant materials when notifying the user.

[0808] Through these steps, the system can extract useful information from users' comments and quickly and accurately notify and share it.

[0809] (Application example 1)

[0810] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0811] In brick-and-mortar stores, it is difficult for staff to respond quickly and accurately to the diverse questions and requests of customers. Furthermore, because of the vast amount of information customers ask, it is unrealistic for staff to keep track of everything. As a result, the quality of customer service may decline, and customer satisfaction may decrease. The present invention aims to provide a system that solves these problems and improves customer service.

[0812] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0813] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, and means for recognizing voice data to collect customer comments in a physical store and displaying related useful information on a smart gadget. This makes it possible to analyze customer comments in real time and provide useful information to staff immediately.

[0814] "User" refers to a person or organization that uses the system.

[0815] "Means for collecting speech" refers to equipment and software for collecting user voice data in real time.

[0816] "Means for converting into text data" refers to speech recognition technology or software for converting collected voice data into character string information.

[0817] "Natural language processing and keyword extraction methods" refers to algorithms and software used to identify and extract important words and phrases from text data.

[0818] "Means for assessing the usefulness of text data" refers to machine learning models and algorithms for assessing the value of information based on extracted keywords.

[0819] "Means of notification" refers to chat tools and applications for smart gadgets that inform users of information that has been rated as highly useful.

[0820] "Brick and mortar store" refers to a physical point of sale where goods or services are sold in person.

[0821] "Means for recognizing voice data" refers to voice recognition technology and devices used to collect and analyze customer speech within a physical store.

[0822] "Smart gadgets" refers to wearable or portable devices that can display information in real time, such as smart glasses or smartphones.

[0823] "Means for displaying relevant useful information" refers to a display and related software for visually providing useful information to the user.

[0824] Overall system overview

[0825] The system according to the present invention comprises a terminal that collects user comments in real time and transmits them as text data to a server, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and notification means that notifies the user of the useful information and provides additional information. A specific embodiment will be described below.

[0826] 1. Data Collection

[0827] The device monitors in real time what users and store staff say during their daily work and conversations, and converts this into text data using voice recognition technology. The device can be a smart eyeglass or a smartphone. A microphone and voice recognition software are built in to accurately analyze the voice data. For example, Microsoft Azure's voice recognition service is used.

[0828] 2. Natural Language Processing (NLP)

[0829] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as Google BERT and OpenAI GPT-3 to identify entities (important words and phrases) within the text.

[0830] 3. Extraction of useful information

[0831] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0832] 4. Notices and Suggestions

[0833] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[0834] 5. Providing Additional Information

[0835] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0836] Specific examples

[0837] For example, if a store staff member is asked by a customer, "Please tell me the stock status of this item," the system works as follows:

[0838] 1. Input prompt example

[0839] Customer: "What is the availability of this item?"

[0840] 2. Audio collection and recognition

[0841] Speech to text: "What is the availability of this item?"

[0842] 3. Natural Language Processing

[0843] Keyword extraction: "Stock status", "Product"

[0844] 4. Usefulness evaluation and notification

[0845] Check the registered product name and availability and provide specific answers such as "In stock" or "Out of stock" to staff

[0846] In this way, store staff can respond to customers in real time, improving the quality of service.The system of the present invention effectively extracts useful information from users' everyday utterances and appropriately notifies and shares it, thereby strengthening the knowledge base of the entire organization.

[0847] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0848] Step 1:

[0849] The device collects the audio data.

[0850] Input: User or customer speech (voice data)

[0851] How it works: The device's microphone collects the user's or customer's voice in real time.

[0852] Output: Collected audio data

[0853] Step 2:

[0854] The terminal converts the voice data into text data.

[0855] Input: Audio data

[0856] How it works: Speech recognition technology (for example, Microsoft Azure's speech recognition service) processes the voice data and converts it into text data.

[0857] Output: Text data

[0858] Step 3:

[0859] Sends text data to the server.

[0860] Input: Text data

[0861] Operation: The device sends the converted text data to a server via the Internet.

[0862] Output: Text data sent to the server

[0863] Step 4:

[0864] The server performs natural language processing on the received text data.

[0865] Input: Text data sent to the server

[0866] How it works: A server-based natural language processing engine (e.g., Google BERT or OpenAI GPT-3) analyzes text data and extracts important keywords and phrases.

[0867] Output: Extracted keywords and phrases

[0868] Step 5:

[0869] The server evaluates the usefulness of the text data based on the extracted keywords.

[0870] Input: Extracted keywords or phrases

[0871] How it works: The server's machine learning model uses historical data and other relevant information to determine the usefulness of the extracted keywords.

[0872] Output: Evaluation result (whether the information is useful or not)

[0873] Step 6:

[0874] The server notifies the user of the text data that has been evaluated as being highly useful.

[0875] Input: Evaluation results (highly useful information)

[0876] Operation: The server generates a notification message and sends it to the user through a notification means (for example, an API for a chat tool).

[0877] Output: Message notified to the user

[0878] Step 7:

[0879] The server will provide additional information as needed.

[0880] Input: Request and context information after user notification

[0881] How it works: The server searches for additional relevant information (e.g., technical documentation or past cases) and provides it to the user.

[0882] Output: Additional information displayed to the user

[0883] Specific examples

[0884] For example, if a customer asks, "Please tell me the stock status of this product," the voice data collected in steps 1-3 is converted into text data and sent to the server. In step 4, natural language processing is performed to extract keywords such as "stock status" and "product." In step 5, the usefulness is evaluated, and if it is determined to be useful, in step 6, the staff member is notified that "This product is in stock." Additional information such as the product's location and price is also provided in step 7.

[0885] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0886] The system according to the present invention collects user comments, evaluates the usefulness and emotions of the comments through natural language processing and an emotion engine, and notifies users appropriately. Specific embodiments for carrying out the present invention will be described below.

[0887] Overall system overview

[0888] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data to extract and evaluate useful information, an emotion engine that recognizes the user's emotions, and a means of notifying the user of the useful information and providing additional information.

[0889] 1. Data Collection

[0890] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[0891] 2. Natural Language Processing (NLP)

[0892] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[0893] 3. Extraction of useful information

[0894] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[0895] 4. Emotion Engine

[0896] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[0897] 5. Notices and Suggestions

[0898] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[0899] 6. Providing Additional Information

[0900] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[0901] Specific examples

[0902] Suppose a user says during a video conference, "This new project is very tough, but I'm working hard." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "tough" are extracted using natural language processing.

[0903] The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the user has the emotion "strict." If the evaluation result is deemed useful, the server notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." If the user is feeling stressed, this notification is further adjusted to include a message of encouragement and support. Additional information is also provided, including success stories of similar projects and documents on efficient project management methods.

[0904] In this way, the system of the present invention can strengthen the knowledge base of the entire organization by efficiently extracting useful insights from users' everyday comments and providing appropriate notifications and sharing that take emotions into consideration.

[0905] The processing flow will be explained below.

[0906] Step 1:

[0907] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[0908] Step 2:

[0909] The device monitors the user's speech in real time, and the monitored voice data is collected by the device's microphone.

[0910] Step 3:

[0911] The device converts the collected voice data into text data using voice recognition technology, such as the Google Speech-to-Text API.

[0912] Step 4:

[0913] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[0914] Step 5:

[0915] The server receives the text data sent from the terminal and temporarily stores it in a database.

[0916] Step 6:

[0917] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[0918] Step 7:

[0919] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[0920] Step 8:

[0921] The server passes the text data to an emotion engine to recognize the user's emotions. The emotion engine detects emotional tones and specific emotion keywords (e.g., joy, anger, sadness) from the text data.

[0922] Step 9:

[0923] The server takes emotional information into account when evaluating the usefulness of text data based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it adjusts the importance of the information.

[0924] Step 10:

[0925] The server notifies the user of information that is rated as highly useful. The notification is sent via the API of the chat tool the user is using.

[0926] Step 11:

[0927] The server adjusts the content and format of notifications based on the emotional assessment: for example, if the user is feeling stressed, notifications will be delivered in a tone of encouragement and support.

[0928] Step 12:

[0929] The server provides the user with additional relevant information as needed, such as detailed documentation of a particular technology or implementation instructions based on past experience.

[0930] Through the above processing steps, this system can effectively extract useful insights from users' everyday comments and strengthen the knowledge base of the entire organization by providing appropriate notifications and sharing that take emotions into consideration.

[0931] Example 2

[0932] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0933] In conventional systems, the process of collecting user comments and evaluating useful information is fragmented, which leads to problems such as inadequate notification and provision of relevant additional information that takes user emotions into account. Furthermore, the accuracy of converting voice data into text is sometimes insufficient, resulting in issues of overall lack of efficiency and accuracy.

[0934] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for identifying the user's emotions from the text data, means for notifying the user when the usefulness is evaluated as high, and means for providing related additional information. This makes it possible to collect user comments in real time, evaluate their usefulness and emotions, and then provide appropriate notifications and information.

[0935] A "means for collecting user utterances" is a device or process that monitors users' voice and text utterances in real time and acquires them as data.

[0936] "Means for converting collected speech into text data" refers to a device or process that converts speech data into text format using natural language processing techniques.

[0937] A "means for performing natural language processing on text data to extract keywords" is a device or process that uses natural language processing techniques to identify important words and phrases within text.

[0938] The "means for evaluating the usefulness of text data based on extracted keywords" refers to a device or process that uses a machine learning model or algorithm to evaluate the usefulness of text based on extracted keywords.

[0939] A "means for identifying user emotions from text data" is a device or process that uses natural language processing technology to analyze the emotional tone and specific emotional keywords in text and identify the user's emotions.

[0940] The "means for notifying the user when the usefulness is evaluated as high" is a device or process that sends an alert or message to the user based on the text data that has been evaluated as being highly useful.

[0941] The "means for providing related additional information" is a device or process that provides a user with additional information, such as documents or examples, related to the evaluated text data.

[0942] The system of the present invention collects user utterances, evaluates the usefulness and sentiment of the utterances through natural language processing and an emotion engine, and provides appropriate notifications and related additional information.

[0943] Overall overview

[0944] The system consists of the following elements:

[0945] Device that collects comments

[0946] Software that converts collected comments into text data

[0947] A server that performs natural language processing on text data to extract and evaluate information

[0948] Emotion engine that recognizes user emotions

[0949] Means of notifying useful information

[0950] A means of providing additional relevant information

[0951] Data collection

[0952] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device uses common communication devices such as PCs and smartphones, and uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data with high accuracy.

[0953] Examples:

[0954] When a user says, "My new project is very challenging, but rewarding," the speech is collected through the device's microphone and converted into text data by speech recognition software.

[0955] Natural Language Processing (NLP)

[0956] The server receives the text data sent from the device and performs natural language processing using large-scale language models such as BERT to extract entities and important phrases within the text.

[0957] Examples:

[0958] The server analyzes the received text data, "The new project is very challenging, but rewarding," and extracts keywords such as "new project," "challenging," and "rewarding."

[0959] Extracting useful information

[0960] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords, and references past case studies and inquiry data stored in the company's database for the evaluation.

[0961] Examples:

[0962] If the keyword "new project" is information that has been evaluated as useful in the past, the current text data will also be evaluated as useful.

[0963] Emotion Engine

[0964] The emotion engine uses natural language processing technology to identify user emotions from text data, detecting emotions such as joy, challenge, and stress from keywords and context.

[0965] Examples:

[0966] From phrases such as "challenging," the user's emotion is identified as "positive challenge."

[0967] Notification of useful information

[0968] The server notifies the user of text data that has been evaluated as highly useful. This notification is made via a chat tool (e.g., Slack API).

[0969] Examples:

[0970] The user receives the message, "This information is important. We recommend that you document it and share it with your team."

[0971] Providing additional relevant information

[0972] The server will provide the user with additional relevant information as needed, including detailed documentation of past cases and technologies.

[0973] Examples:

[0974] Provide users with a link that says, "Here is detailed documentation of past success stories."

[0975] Prompt Sentence Examples

[0976] "The new project is very challenging. Please rate the usefulness and sentiment of this statement. Also, please provide any additional relevant information."

[0977] According to this invention, it is possible to collect and analyze user comments in real time and evaluate the usefulness and sentiment of the comments, thereby making it possible to provide appropriate notifications and information.

[0978] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0979] Step 1: Data collection

[0980] The device monitors the user's speech in real time and collects it as voice data. At this stage, the input is the user's speech into the microphone. The voice data is passed to the device's voice recognition software (e.g., Google Speech-to-Text API). The voice recognition software analyzes the voice data and converts it into text data. The output is the converted text data.

[0981] Specifically, the user says, "This new project is very challenging, but worthwhile." This is collected as voice data on the device.

[0982] Step 2: Sending text data

[0983] The device sends the text data converted from the voice data to the server. The input is the text data obtained in the previous step. The device sends this text data to the server using a RESTful API. The output is the text data received by the server.

[0984] Specifically, the text data "This new project is very challenging, but rewarding" is sent.

[0985] Step 3: Natural Language Processing (NLP)

[0986] The server analyzes the text data received from the device and extracts keywords and important phrases using natural language processing techniques. The input is the received text data. The server uses a large-scale language model such as BERT to identify important entities within the text. The output is the extracted keywords and phrases.

[0987] Specifically, the server extracts the keywords "new project," "challenging," and "rewarding" from the text "This new project is very challenging, but rewarding."

[0988] Step 4: Extract useful information

[0989] The server evaluates the usefulness of the text data based on the extracted keywords and phrases. The input is the extracted keywords. Using a machine learning model, the evaluation is performed by referencing past cases and inquiry data from an internal database. The output is a usefulness score for the text data.

[0990] As a concrete example, let's say the keyword "new project" has been frequently used in past highly rated cases. In this case, the text data will be evaluated as useful.

[0991] Step 5: Sentiment analysis

[0992] The server uses an emotion engine to analyze user emotions from text data. The input is text data. Natural language processing techniques are used to detect context and specific emotion keywords and identify emotions. The output is the detected emotion information.

[0993] As a specific operation, it identifies that the user has a positive feeling of challenge from the word "challenging."

[0994] Step 6: Notification

[0995] The server notifies the user based on the text data that was rated as highly useful and the results of sentiment analysis. The input is the usefulness score and sentiment information of the text data. The notification is sent using the chat tool's API (e.g., Slack API). The output is a notification message sent to the user.

[0996] Specifically, the system sends a message to the user saying, "This information is important. We recommend that you document it and share it with your team." Based on the sentiment, it also adds words of encouragement.

[0997] Step 7: Provide additional information

[0998] The server provides the user with additional related information as needed. The input is keywords in the text data and past case data. Related documents and case links are generated and provided. The output is the additional information provided to the user.

[0999] Specifically, the user is provided with a link that says, "Click here for detailed documentation on past success stories."

[1000] Through these steps, the system collects and analyzes user comments in real time, evaluates their usefulness and sentiment, and provides appropriate notifications and additional information.

[1001] (Application example 2)

[1002] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1003] In the past, customer service in brick-and-mortar stores did not analyze communication between customers and staff in real time, making it difficult to accurately grasp customer needs and emotions. Furthermore, there was a lack of a system for providing appropriate feedback and suggestions quickly, making it difficult to respond immediately to improve customer satisfaction. For this reason, there was a need for a way to improve staff response efficiency and increase customer satisfaction.

[1004] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1005] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, means for analyzing the user's emotions from the text data and adjusting the content of the notification based on the emotions, and means for providing related additional information under specific conditions. This makes it possible to analyze communication between customers and staff in real time and provide quick and appropriate feedback and suggestions based on the customer's needs and emotions.

[1006] "Means for collecting user utterances" refers to a voice input device for collecting voice conversations and utterances made by customers and staff in real time. Typically, this is done using a microphone built into smart glasses or a smartphone.

[1007] "Means for converting collected speech into text data" refers to technology that converts collected speech data into text data using speech recognition technology, including the Google Speech Recognition API and the SpeechRecognition library.

[1008] "Means of extracting keywords by applying natural language processing to text data" refers to natural language processing techniques that analyze text data and extract important words and phrases. Large-scale language models such as BERT and Transformers are used.

[1009] "Means for evaluating the usefulness of text data based on extracted keywords" refers to a machine learning model that analyzes extracted keywords and determines whether the text data is useful. Criteria used include internal historical data and entity frequency.

[1010] "Means for notifying users when information is rated as highly useful" refers to a means for notifying customers and staff of information that has been rated as useful. This can be done using the API of the chat application or a notification system.

[1011] "Means of analyzing user emotions from text data and adjusting notification content based on those emotions" refers to technology that analyzes the emotions of customers and staff from text data and appropriately adjusts the content and format of notifications based on the analysis results. Emotion analysis utilizes an emotion recognition pipeline.

[1012] "Means for providing relevant additional information under specific conditions" refers to means for providing relevant information that a user needs. This is a system that suggests additional information such as past cases, detailed documentation, and implementation methods.

[1013] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and an emotion engine that recognizes the user's emotions.It also includes a means to appropriately notify the user of useful information and provide additional information.

[1014] 1. Data Collection

[1015] The terminals, which are smart glasses or smartphones equipped with microphones and voice recognition software, monitor everyday conversations between customers and staff in real time and convert them into text using voice recognition technology.

[1016] 2. Natural Language Processing (NLP)

[1017] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This analysis involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as BERT to identify entities (important words and phrases) within the text.

[1018] 3. Extraction of useful information

[1019] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information. This evaluation is based on past data and the frequency of entities, and determines the usefulness of the information.

[1020] 4. Emotion Engine

[1021] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[1022] 5. Notices and Suggestions

[1023] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[1024] 6. Providing Additional Information

[1025] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[1026] Specific examples

[1027] Suppose a customer says to a staff member wearing smart glasses, "This new product is a little difficult to use." This utterance is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new product" and "difficult to use" are extracted through natural language processing. The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the customer has negative emotions. If the evaluation result is deemed useful, the staff member is notified, saying, "It seems that the customer is dissatisfied with the new product. Please listen to their story in more detail and make suggestions to improve usability." An example of a prompt sentence is as follows:

[1028] Prompt Sentence Examples

[1029] "Japanese text data: 'This new product is a little difficult to use.' Please analyze it with a sentiment analysis tool."

[1030] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1031] Step 1:

[1032] The terminal monitors everyday conversations between customers and staff in real time and collects voice data. The collected voice data is captured using a microphone built into smart glasses or smartphones. The input is voice data, and the output is an audio file containing the recorded voice data.

[1033] Step 2:

[1034] The device converts the collected voice data into text data using voice recognition technology. The Google speech recognition API and SpeechRecognition library are used here. The input is voice data, and the output is text data converted from the voice data.

[1035] Step 3:

[1036] The server receives text data sent from the terminal. The input is text data, and the output is also text data. This step mainly involves data transfer.

[1037] Step 4:

[1038] The server performs natural language processing (NLP) on the received text data to extract keywords. Here, generative AI models such as BERT and Transformers are used to analyze sentences and identify important entities (words and phrases). The input is the text data, and the output is the extracted keywords.

[1039] Step 5:

[1040] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords. The evaluation uses criteria such as past data and entity frequency. The input is the extracted keywords, and the output is a score indicating the usefulness of the text data.

[1041] Step 6:

[1042] The server's emotion engine analyzes user emotions from text data. It uses an emotion recognition pipeline to detect emotional tones and specific emotion keywords (e.g., joy, anger, sadness) in the text. The input is text data, and the output is the analyzed emotion information.

[1043] Step 7:

[1044] The server notifies the user of text data that is rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is performed using the API of a chat tool. The input is the usefulness score and sentiment information of the text data, and the output is a notification message to the user.

[1045] Step 8:

[1046] The server adjusts the notification content and format based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will send a suggested message in a softer tone. This adjustment is made using the results of emotion analysis. The input is emotional information, and the output is the adjusted notification message.

[1047] Step 9:

[1048] The server provides users with additional information related to a specific condition, such as detailed documentation on a specific technology or implementation methods based on past examples. The input is text data evaluated as highly useful, and the output is the related additional information.

[1049] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1050] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1051] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1052] [Fourth embodiment]

[1053] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1054] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1055] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1056] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1057] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1058] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1059] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1060] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1061] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1062] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1063] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1064] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1065] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1066] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[1067] Overall system overview

[1068] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[1069] 1. Data Collection

[1070] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[1071] 2. Natural Language Processing (NLP)

[1072] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[1073] 3. Extraction of useful information

[1074] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[1075] 4. Notices and Suggestions

[1076] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[1077] 5. Providing Additional Information

[1078] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[1079] Specific examples

[1080] Suppose a user says during a video conference, "Please report on the progress of the new project." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing.

[1081] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[1082] In this way, the system according to the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[1083] The processing flow will be explained below.

[1084] Step 1:

[1085] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[1086] Step 2:

[1087] The device monitors the user's speech in real time, sometimes using speech recognition technology to collect speech data, and sometimes collecting it directly as text data.

[1088] Step 3:

[1089] The device converts the collected voice data into text data using voice recognition technology, such as voice recognition software like the Google Speech-to-Text API.

[1090] Step 4:

[1091] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[1092] Step 5:

[1093] The server receives the text data sent from the terminal and temporarily stores it in a database.

[1094] Step 6:

[1095] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[1096] Step 7:

[1097] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[1098] Step 8:

[1099] The server notifies the user of information that has been evaluated as highly useful. The notification is made, for example, by sending a message to the user's device using the API of a chat tool.

[1100] Step 9:

[1101] The server suggests that useful information is contained in the notification message and that this information be documented and shared. The notification may also include specific instructions and suggested documentation formats.

[1102] Step 10:

[1103] The server provides additional relevant information as needed, such as detailed documentation of a specific technology or implementation methods based on past examples, for user reference.

[1104] In this way, the system effectively extracts useful insights from users' everyday comments and realizes the function of notifying and sharing them at the appropriate time.

[1105] Example 1

[1106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1107] Conventional voice data collection and analysis systems lack the functionality to extract useful information from user speech in real time and appropriately notify and share it. This can result in important information being overlooked, making it difficult to contribute to the organization's overall knowledge base. Furthermore, there are issues with the accuracy of converting voice data into text data and the accuracy of analysis using natural language processing technology. There is a need for a system that can resolve these issues, quickly and accurately extract useful information from user speech, and share it within an organization.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1109] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for transmitting the text data to a central processing unit via a communication system, means for extracting keywords from the text data using natural language analysis technology, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user based on the evaluated usefulness, and means for providing related additional information. This makes it possible to quickly and accurately extract useful information from user comments and to appropriately notify and share it.

[1110] "Means for collecting user utterances" refers to devices or software for capturing voice and text data uttered by users.

[1111] The "means of converting collected speech into text data" refers to the process of converting speech data into text data using speech recognition technology or a conversion algorithm.

[1112] "Means for transmitting text data to a central processing unit via a communication system" refers to protocols or means for transmitting converted text data to a central processing unit such as a server or cloud using a communication network.

[1113] "Natural language analysis technology" is a technology for analyzing text data and understanding its content and context, and includes algorithms for extracting keywords and important phrases.

[1114] "Keyword extraction" is the process of using natural language processing techniques to identify important words and phrases from text data.

[1115] "Means for assessing the usefulness of text data" refers to the process of using machine learning models or algorithms to assess the value or importance of the text data based on extracted keywords.

[1116] "Means for notifying the user" refers to a method for communicating the evaluated information to the user, and includes email, chat tools, push notifications, etc.

[1117] The "means for providing additional relevant information" is a process of providing additional resources or documents that are useful to the user based on the evaluated text data.

[1118] The system according to the present invention effectively collects user comments, automatically detects useful information and insights through natural language processing, notifies the user of the information, and further shares the information within an organization. Specific embodiments for carrying out the present invention will be described below.

[1119] System hardware and software configuration

[1120] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and a notification means that notifies the user of this useful information and provides additional information.

[1121] Terminal

[1122] The device monitors the user's daily work and conversations in real time and converts them into text data using voice recognition technology. The device may be the user's personal computer or smartphone, and is equipped with a microphone and voice recognition software (e.g., Google Speech-to-Text or Microsoft Azure Speech Recognition) for highly accurate analysis of the voice data.

[1123] server

[1124] The server receives the text data sent from the device and analyzes it using natural language processing techniques (e.g., large-scale language models such as BERT or GPT-4). This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Furthermore, based on the extracted entity information, the server evaluates the usefulness of the text data using a machine learning model (e.g., a custom model using Scikit-learn or TensorFlow).

[1125] Notification means

[1126] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is realized, for example, using the API of the chat tool (e.g., Slack API or Microsoft Teams API). If necessary, additional related information (e.g., detailed documentation on a specific technology or implementation methods based on past examples) is also provided to the user.

[1127] Specific examples

[1128] Consider a scenario where a user says, "Please report on the progress of the new project" during a video conference. This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "progress" are extracted using natural language processing technology.

[1129] The server evaluates the usefulness of the text data based on these keywords, and if it determines that similar information from the past is important, it notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." It also provides additional information, such as specific report formats for project progress and success stories from past projects.

[1130] Prompt Sentence Examples

[1131] Below are some example prompts to input to a generative AI model:

[1132] > "Please collect what users say during video conferences in real time, convert it into text data, and send it to a server. Then, please explain in detail how the system will analyze the text data using natural language processing technology, extract useful information, and notify users. Please also include the names of any specific hardware or software."

[1133] As a result, the present invention can effectively extract useful information from users' everyday utterances and appropriately notify and share it, thereby strengthening the knowledge base of the entire organization.

[1134] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1135] Step 1: Data collection

[1136] The device monitors the user's speech in real time and collects audio data. The input is the user's voice, and once acquired, it is saved as audio data. Specifically, the device's microphone captures the audio and saves it as an audio file.

[1137] Step 2: Voice Recognition

[1138] The device converts the voice data collected in step 1 into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it calls the Google Speech-to-Text or Microsoft Azure Speech Recognition API to send voice data and receive text data.

[1139] Step 3: Send text data

[1140] The terminal sends the text data generated in step 2 to the server. The input is text data, and the output is text data sent over the network. Specifically, the text data is sent to the server using the HTTPS protocol.

[1141] Step 4: Receiving and saving text data

[1142] The server receives and stores the text data sent in step 3. The input is the text data received from the network, and the output is the text data stored in the server. Specifically, the server stores the received text data in a database.

[1143] Step 5: Natural Language Processing (NLP)

[1144] The server performs natural language analysis on the stored text data. The input is the text data, and the output is extracted keywords and phrases. Specifically, it uses the BERT model to analyze the text data and extract important entities.

[1145] Step 6: Usability evaluation

[1146] The server evaluates the usefulness of the text data based on the keywords and phrases extracted in step 5. The input is the keywords and phrases, and the output is a usefulness evaluation score. Specifically, it applies a machine learning model using Scikit-learn and calculates the usefulness score by referring to past data and frequency data.

[1147] Step 7: User Notification

[1148] The server notifies the user based on the text data that was rated as highly useful. The input is the usefulness rating score and the rated text data, and the output is a notification to the user. Specifically, it uses the API of the chat tool (e.g., Slack API) to send the rating result and a message to the user saying, "This information is important. We recommend that you document it and share it with your team."

[1149] Step 8: Provide additional information

[1150] The server provides the user with additional relevant information as needed. The input is the evaluated text data, and the output is additional information such as related documents and past cases. Specifically, the server searches the knowledge base within the server and attaches relevant materials when notifying the user.

[1151] Through these steps, the system can extract useful information from users' comments and quickly and accurately notify and share it.

[1152] (Application example 1)

[1153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1154] In brick-and-mortar stores, it is difficult for staff to respond quickly and accurately to the diverse questions and requests of customers. Furthermore, because of the vast amount of information customers ask, it is unrealistic for staff to keep track of everything. As a result, the quality of customer service may decline, and customer satisfaction may decrease. The present invention aims to provide a system that solves these problems and improves customer service.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1156] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, and means for recognizing voice data to collect customer comments in a physical store and displaying related useful information on a smart gadget. This makes it possible to analyze customer comments in real time and provide useful information to staff immediately.

[1157] "User" refers to a person or organization that uses the system.

[1158] "Means for collecting speech" refers to equipment and software for collecting user voice data in real time.

[1159] "Means for converting into text data" refers to speech recognition technology or software for converting collected voice data into character string information.

[1160] "Natural language processing and keyword extraction methods" refers to algorithms and software used to identify and extract important words and phrases from text data.

[1161] "Means for assessing the usefulness of text data" refers to machine learning models and algorithms for assessing the value of information based on extracted keywords.

[1162] "Means of notification" refers to chat tools and applications for smart gadgets that inform users of information that has been rated as highly useful.

[1163] "Brick and mortar store" refers to a physical point of sale where goods or services are sold in person.

[1164] "Means for recognizing voice data" refers to voice recognition technology and devices used to collect and analyze customer speech within a physical store.

[1165] "Smart gadgets" refers to wearable or portable devices that can display information in real time, such as smart glasses or smartphones.

[1166] "Means for displaying relevant useful information" refers to a display and related software for visually providing useful information to the user.

[1167] Overall system overview

[1168] The system according to the present invention comprises a terminal that collects user comments in real time and transmits them as text data to a server, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and notification means that notifies the user of the useful information and provides additional information. A specific embodiment will be described below.

[1169] 1. Data Collection

[1170] The device monitors in real time what users and store staff say during their daily work and conversations, and converts this into text data using voice recognition technology. The device can be a smart eyeglass or a smartphone. A microphone and voice recognition software are built in to accurately analyze the voice data. For example, Microsoft Azure's voice recognition service is used.

[1171] 2. Natural Language Processing (NLP)

[1172] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as Google BERT and OpenAI GPT-3 to identify entities (important words and phrases) within the text.

[1173] 3. Extraction of useful information

[1174] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[1175] 4. Notices and Suggestions

[1176] The server notifies users of text data that has been rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification can be realized, for example, using the API of a chat tool.

[1177] 5. Providing Additional Information

[1178] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[1179] Specific examples

[1180] For example, if a store staff member is asked by a customer, "Please tell me the stock status of this item," the system works as follows:

[1181] 1. Input prompt example

[1182] Customer: "What is the availability of this item?"

[1183] 2. Audio collection and recognition

[1184] Speech to text: "What is the availability of this item?"

[1185] 3. Natural Language Processing

[1186] Keyword extraction: "Stock status", "Product"

[1187] 4. Usefulness evaluation and notification

[1188] Check the registered product name and availability and provide specific answers such as "In stock" or "Out of stock" to staff

[1189] In this way, store staff can respond to customers in real time, improving the quality of service.The system of the present invention effectively extracts useful information from users' everyday utterances and appropriately notifies and shares it, thereby strengthening the knowledge base of the entire organization.

[1190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1191] Step 1:

[1192] The device collects the audio data.

[1193] Input: User or customer speech (voice data)

[1194] How it works: The device's microphone collects the user's or customer's voice in real time.

[1195] Output: Collected audio data

[1196] Step 2:

[1197] The terminal converts the voice data into text data.

[1198] Input: Audio data

[1199] How it works: Speech recognition technology (for example, Microsoft Azure's speech recognition service) processes the voice data and converts it into text data.

[1200] Output: Text data

[1201] Step 3:

[1202] Sends text data to the server.

[1203] Input: Text data

[1204] Operation: The device sends the converted text data to a server via the Internet.

[1205] Output: Text data sent to the server

[1206] Step 4:

[1207] The server performs natural language processing on the received text data.

[1208] Input: Text data sent to the server

[1209] How it works: A server-based natural language processing engine (e.g., Google BERT or OpenAI GPT-3) analyzes text data and extracts important keywords and phrases.

[1210] Output: Extracted keywords and phrases

[1211] Step 5:

[1212] The server evaluates the usefulness of the text data based on the extracted keywords.

[1213] Input: Extracted keywords or phrases

[1214] How it works: The server's machine learning model uses historical data and other relevant information to determine the usefulness of the extracted keywords.

[1215] Output: Evaluation result (whether the information is useful or not)

[1216] Step 6:

[1217] The server notifies the user of the text data that has been evaluated as being highly useful.

[1218] Input: Evaluation results (highly useful information)

[1219] Operation: The server generates a notification message and sends it to the user through a notification means (for example, an API for a chat tool).

[1220] Output: Message notified to the user

[1221] Step 7:

[1222] The server will provide additional information as needed.

[1223] Input: Request and context information after user notification

[1224] How it works: The server searches for additional relevant information (e.g., technical documentation or past cases) and provides it to the user.

[1225] Output: Additional information displayed to the user

[1226] Specific examples

[1227] For example, if a customer asks, "Please tell me the stock status of this product," the voice data collected in steps 1-3 is converted into text data and sent to the server. In step 4, natural language processing is performed to extract keywords such as "stock status" and "product." In step 5, the usefulness is evaluated, and if it is determined to be useful, in step 6, the staff member is notified that "This product is in stock." Additional information such as the product's location and price is also provided in step 7.

[1228] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1229] The system according to the present invention collects user comments, evaluates the usefulness and emotions of the comments through natural language processing and an emotion engine, and notifies users appropriately. Specific embodiments for carrying out the present invention will be described below.

[1230] Overall system overview

[1231] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data to extract and evaluate useful information, an emotion engine that recognizes the user's emotions, and a means of notifying the user of the useful information and providing additional information.

[1232] 1. Data Collection

[1233] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device, which can be a user's PC or smartphone, is equipped with a microphone and voice recognition software for highly accurate analysis of the voice data.

[1234] 2. Natural Language Processing (NLP)

[1235] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses large-scale language models such as BERT to identify entities (important words and phrases) within the text.

[1236] 3. Extraction of useful information

[1237] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information, taking into account data such as how useful similar information has been in the past and how frequently it has been asked about within the company.

[1238] 4. Emotion Engine

[1239] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[1240] 5. Notices and Suggestions

[1241] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[1242] 6. Providing Additional Information

[1243] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[1244] Specific examples

[1245] Suppose a user says during a video conference, "This new project is very tough, but I'm working hard." This speech is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new project" and "tough" are extracted using natural language processing.

[1246] The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the user has the emotion "strict." If the evaluation result is deemed useful, the server notifies the user, saying, "This information is important. We recommend that you document it and share it with your team." If the user is feeling stressed, this notification is further adjusted to include a message of encouragement and support. Additional information is also provided, including success stories of similar projects and documents on efficient project management methods.

[1247] In this way, the system of the present invention can strengthen the knowledge base of the entire organization by efficiently extracting useful insights from users' everyday comments and providing appropriate notifications and sharing that take emotions into consideration.

[1248] The processing flow will be explained below.

[1249] Step 1:

[1250] Users make statements in their daily work and conversations, for example, they make specific statements such as "This bug can be found in the log file."

[1251] Step 2:

[1252] The device monitors the user's speech in real time, and the monitored voice data is collected by the device's microphone.

[1253] Step 3:

[1254] The device converts the collected voice data into text data using voice recognition technology, such as the Google Speech-to-Text API.

[1255] Step 4:

[1256] The terminal transmits the converted text data to the server, using a secure communication protocol (e.g., HTTPS) to ensure the safety of the data.

[1257] Step 5:

[1258] The server receives the text data sent from the terminal and temporarily stores it in a database.

[1259] Step 6:

[1260] The server performs natural language processing on the received text data, specifically using the BERT model and other NLP libraries to analyze the meaning of the sentences and extract keywords and phrases.

[1261] Step 7:

[1262] The server evaluates the usefulness of the text data based on the extracted keywords and phrases, using a machine learning model to compare it with a historical database to determine how useful the information is.

[1263] Step 8:

[1264] The server passes the text data to an emotion engine to recognize the user's emotions. The emotion engine detects emotional tones and specific emotion keywords (e.g., joy, anger, sadness) from the text data.

[1265] Step 9:

[1266] The server takes emotional information into account when evaluating the usefulness of text data based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it adjusts the importance of the information.

[1267] Step 10:

[1268] The server notifies the user of information that is rated as highly useful. The notification is sent via the API of the chat tool the user is using.

[1269] Step 11:

[1270] The server adjusts the content and format of notifications based on the emotional assessment: for example, if the user is feeling stressed, notifications will be delivered in a tone of encouragement and support.

[1271] Step 12:

[1272] The server provides the user with additional relevant information as needed, such as detailed documentation of a particular technology or implementation instructions based on past experience.

[1273] Through the above processing steps, this system can effectively extract useful insights from users' everyday comments and strengthen the knowledge base of the entire organization by providing appropriate notifications and sharing that take emotions into consideration.

[1274] Example 2

[1275] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1276] In conventional systems, the process of collecting user comments and evaluating useful information is fragmented, which leads to problems such as inadequate notification and provision of relevant additional information that takes user emotions into account. Furthermore, the accuracy of converting voice data into text is sometimes insufficient, resulting in issues of overall lack of efficiency and accuracy.

[1277] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for identifying the user's emotions from the text data, means for notifying the user when the usefulness is evaluated as high, and means for providing related additional information. This makes it possible to collect user comments in real time, evaluate their usefulness and emotions, and then provide appropriate notifications and information.

[1278] A "means for collecting user utterances" is a device or process that monitors users' voice and text utterances in real time and acquires them as data.

[1279] "Means for converting collected speech into text data" refers to a device or process that converts speech data into text format using natural language processing techniques.

[1280] A "means for performing natural language processing on text data to extract keywords" is a device or process that uses natural language processing techniques to identify important words and phrases within text.

[1281] The "means for evaluating the usefulness of text data based on extracted keywords" refers to a device or process that uses a machine learning model or algorithm to evaluate the usefulness of text based on extracted keywords.

[1282] A "means for identifying user emotions from text data" is a device or process that uses natural language processing technology to analyze the emotional tone and specific emotional keywords in text and identify the user's emotions.

[1283] The "means for notifying the user when the usefulness is evaluated as high" is a device or process that sends an alert or message to the user based on the text data that has been evaluated as being highly useful.

[1284] The "means for providing related additional information" is a device or process that provides a user with additional information, such as documents or examples, related to the evaluated text data.

[1285] The system of the present invention collects user utterances, evaluates the usefulness and sentiment of the utterances through natural language processing and an emotion engine, and provides appropriate notifications and related additional information.

[1286] Overall overview

[1287] The system consists of the following elements:

[1288] Device that collects comments

[1289] Software that converts collected comments into text data

[1290] A server that performs natural language processing on text data to extract and evaluate information

[1291] Emotion engine that recognizes user emotions

[1292] Means of notifying useful information

[1293] A means of providing additional relevant information

[1294] Data collection

[1295] The device monitors what users say in their daily work and conversations in real time and converts it into text data using voice recognition technology. The device uses common communication devices such as PCs and smartphones, and uses voice recognition software (e.g., Google Speech-to-Text API) to convert the collected voice data into text data with high accuracy.

[1296] Examples:

[1297] When a user says, "My new project is very challenging, but rewarding," the speech is collected through the device's microphone and converted into text data by speech recognition software.

[1298] Natural Language Processing (NLP)

[1299] The server receives the text data sent from the device and performs natural language processing using large-scale language models such as BERT to extract entities and important phrases within the text.

[1300] Examples:

[1301] The server analyzes the received text data, "The new project is very challenging, but rewarding," and extracts keywords such as "new project," "challenging," and "rewarding."

[1302] Extracting useful information

[1303] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords, and references past case studies and inquiry data stored in the company's database for the evaluation.

[1304] Examples:

[1305] If the keyword "new project" is information that has been evaluated as useful in the past, the current text data will also be evaluated as useful.

[1306] Emotion Engine

[1307] The emotion engine uses natural language processing technology to identify user emotions from text data, detecting emotions such as joy, challenge, and stress from keywords and context.

[1308] Examples:

[1309] From phrases such as "challenging," the user's emotion is identified as "positive challenge."

[1310] Notification of useful information

[1311] The server notifies the user of text data that has been evaluated as highly useful. This notification is made via a chat tool (e.g., Slack API).

[1312] Examples:

[1313] The user receives the message, "This information is important. We recommend that you document it and share it with your team."

[1314] Providing additional relevant information

[1315] The server will provide the user with additional relevant information as needed, including detailed documentation of past cases and technologies.

[1316] Examples:

[1317] Provide users with a link that says, "Here is detailed documentation of past success stories."

[1318] Prompt Sentence Examples

[1319] "The new project is very challenging. Please rate the usefulness and sentiment of this statement. Also, please provide any additional relevant information."

[1320] According to this invention, it is possible to collect and analyze user comments in real time and evaluate the usefulness and sentiment of the comments, thereby making it possible to provide appropriate notifications and information.

[1321] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1322] Step 1: Data collection

[1323] The device monitors the user's speech in real time and collects it as voice data. At this stage, the input is the user's speech into the microphone. The voice data is passed to the device's voice recognition software (e.g., Google Speech-to-Text API). The voice recognition software analyzes the voice data and converts it into text data. The output is the converted text data.

[1324] Specifically, the user says, "This new project is very challenging, but worthwhile." This is collected as voice data on the device.

[1325] Step 2: Sending text data

[1326] The device sends the text data converted from the voice data to the server. The input is the text data obtained in the previous step. The device sends this text data to the server using a RESTful API. The output is the text data received by the server.

[1327] Specifically, the text data "This new project is very challenging, but rewarding" is sent.

[1328] Step 3: Natural Language Processing (NLP)

[1329] The server analyzes the text data received from the device and extracts keywords and important phrases using natural language processing techniques. The input is the received text data. The server uses a large-scale language model such as BERT to identify important entities within the text. The output is the extracted keywords and phrases.

[1330] Specifically, the server extracts the keywords "new project," "challenging," and "rewarding" from the text "This new project is very challenging, but rewarding."

[1331] Step 4: Extract useful information

[1332] The server evaluates the usefulness of the text data based on the extracted keywords and phrases. The input is the extracted keywords. Using a machine learning model, the evaluation is performed by referencing past cases and inquiry data from an internal database. The output is a usefulness score for the text data.

[1333] As a concrete example, let's say the keyword "new project" has been frequently used in past highly rated cases. In this case, the text data will be evaluated as useful.

[1334] Step 5: Sentiment analysis

[1335] The server uses an emotion engine to analyze user emotions from text data. The input is text data. Natural language processing techniques are used to detect context and specific emotion keywords and identify emotions. The output is the detected emotion information.

[1336] As a specific operation, it identifies that the user has a positive feeling of challenge from the word "challenging."

[1337] Step 6: Notification

[1338] The server notifies the user based on the text data that was rated as highly useful and the results of sentiment analysis. The input is the usefulness score and sentiment information of the text data. The notification is sent using the chat tool's API (e.g., Slack API). The output is a notification message sent to the user.

[1339] Specifically, the system sends a message to the user saying, "This information is important. We recommend that you document it and share it with your team." Based on the sentiment, it also adds words of encouragement.

[1340] Step 7: Provide additional information

[1341] The server provides the user with additional related information as needed. The input is keywords in the text data and past case data. Related documents and case links are generated and provided. The output is the additional information provided to the user.

[1342] Specifically, the user is provided with a link that says, "Click here for detailed documentation on past success stories."

[1343] Through these steps, the system collects and analyzes user comments in real time, evaluates their usefulness and sentiment, and provides appropriate notifications and additional information.

[1344] (Application example 2)

[1345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1346] In the past, customer service in brick-and-mortar stores did not analyze communication between customers and staff in real time, making it difficult to accurately grasp customer needs and emotions. Furthermore, there was a lack of a system for providing appropriate feedback and suggestions quickly, making it difficult to respond immediately to improve customer satisfaction. For this reason, there was a need for a way to improve staff response efficiency and increase customer satisfaction.

[1347] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1348] In this invention, the server includes means for collecting user comments, means for converting the collected comments into text data, means for performing natural language processing on the text data to extract keywords, means for evaluating the usefulness of the text data based on the extracted keywords, means for notifying the user when the text data is evaluated as being highly useful, means for analyzing the user's emotions from the text data and adjusting the content of the notification based on the emotions, and means for providing related additional information under specific conditions. This makes it possible to analyze communication between customers and staff in real time and provide quick and appropriate feedback and suggestions based on the customer's needs and emotions.

[1349] "Means for collecting user utterances" refers to a voice input device for collecting voice conversations and utterances made by customers and staff in real time. Typically, this is done using a microphone built into smart glasses or a smartphone.

[1350] "Means for converting collected speech into text data" refers to technology that converts collected speech data into text data using speech recognition technology, including the Google Speech Recognition API and the SpeechRecognition library.

[1351] "Means of extracting keywords by applying natural language processing to text data" refers to natural language processing techniques that analyze text data and extract important words and phrases. Large-scale language models such as BERT and Transformers are used.

[1352] "Means for evaluating the usefulness of text data based on extracted keywords" refers to a machine learning model that analyzes extracted keywords and determines whether the text data is useful. Criteria used include internal historical data and entity frequency.

[1353] "Means for notifying users when information is rated as highly useful" refers to a means for notifying customers and staff of information that has been rated as useful. This can be done using the API of the chat application or a notification system.

[1354] "Means of analyzing user emotions from text data and adjusting notification content based on those emotions" refers to technology that analyzes the emotions of customers and staff from text data and appropriately adjusts the content and format of notifications based on the analysis results. Emotion analysis utilizes an emotion recognition pipeline.

[1355] "Means for providing relevant additional information under specific conditions" refers to means for providing relevant information that a user needs. This is a system that suggests additional information such as past cases, detailed documentation, and implementation methods.

[1356] This system consists of a terminal that collects user comments in real time and sends them to a server as text data, a server that performs natural language processing on the received text data, extracts and evaluates useful information, and an emotion engine that recognizes the user's emotions.It also includes a means to appropriately notify the user of useful information and provide additional information.

[1357] 1. Data Collection

[1358] The terminals, which are smart glasses or smartphones equipped with microphones and voice recognition software, monitor everyday conversations between customers and staff in real time and convert them into text using voice recognition technology.

[1359] 2. Natural Language Processing (NLP)

[1360] The server receives the text data sent from the device and analyzes it using natural language processing techniques. This analysis involves understanding the meaning of the sentence and extracting specific keywords and phrases. Specifically, it uses generative AI models such as BERT to identify entities (important words and phrases) within the text.

[1361] 3. Extraction of useful information

[1362] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted entity information. This evaluation is based on past data and the frequency of entities, and determines the usefulness of the information.

[1363] 4. Emotion Engine

[1364] The emotion engine identifies user emotions from text data, for example by using natural language processing techniques to detect emotional tones and specific emotion keywords (such as joy, anger, sadness, etc.) in the text, allowing for understanding the emotions behind statements and utilizing them for evaluation.

[1365] 5. Notices and Suggestions

[1366] The server notifies the user of text data that is rated as highly useful. The notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is implemented, for example, using the API of a chat tool. The content and format of the notification can also be adjusted based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, a suggestion message in a softer tone can be sent.

[1367] 6. Providing Additional Information

[1368] The server provides users with additional relevant information as needed, such as detailed documentation of specific technologies or implementation methods based on past examples, allowing users to further enrich the information they already have.

[1369] Specific examples

[1370] Suppose a customer says to a staff member wearing smart glasses, "This new product is a little difficult to use." This utterance is collected as audio through the device's microphone and converted into text data using speech recognition technology. The text data is sent to a server, where keywords such as "new product" and "difficult to use" are extracted through natural language processing. The server evaluates the usefulness of the text data based on these keywords and uses an emotion engine to recognize that the customer has negative emotions. If the evaluation result is deemed useful, the staff member is notified, saying, "It seems that the customer is dissatisfied with the new product. Please listen to their story in more detail and make suggestions to improve usability." An example of a prompt sentence is as follows:

[1371] Prompt Sentence Examples

[1372] "Japanese text data: 'This new product is a little difficult to use.' Please analyze it with a sentiment analysis tool."

[1373] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1374] Step 1:

[1375] The terminal monitors everyday conversations between customers and staff in real time and collects voice data. The collected voice data is captured using a microphone built into smart glasses or smartphones. The input is voice data, and the output is an audio file containing the recorded voice data.

[1376] Step 2:

[1377] The device converts the collected voice data into text data using voice recognition technology. The Google speech recognition API and SpeechRecognition library are used here. The input is voice data, and the output is text data converted from the voice data.

[1378] Step 3:

[1379] The server receives text data sent from the terminal. The input is text data, and the output is also text data. This step mainly involves data transfer.

[1380] Step 4:

[1381] The server performs natural language processing (NLP) on the received text data to extract keywords. Here, generative AI models such as BERT and Transformers are used to analyze sentences and identify important entities (words and phrases). The input is the text data, and the output is the extracted keywords.

[1382] Step 5:

[1383] The server uses a machine learning model to evaluate the usefulness of the text data based on the extracted keywords. The evaluation uses criteria such as past data and entity frequency. The input is the extracted keywords, and the output is a score indicating the usefulness of the text data.

[1384] Step 6:

[1385] The server's emotion engine analyzes user emotions from text data. It uses an emotion recognition pipeline to detect emotional tones and specific emotion keywords (e.g., joy, anger, sadness) in the text. The input is text data, and the output is the analyzed emotion information.

[1386] Step 7:

[1387] The server notifies the user of text data that is rated as highly useful. This notification includes a message that the text data is useful and suggests documenting and sharing the information. The notification is performed using the API of a chat tool. The input is the usefulness score and sentiment information of the text data, and the output is a notification message to the user.

[1388] Step 8:

[1389] The server adjusts the notification content and format based on the evaluation results of the emotion engine. For example, if the user is feeling stressed, it will send a suggested message in a softer tone. This adjustment is made using the results of emotion analysis. The input is emotional information, and the output is the adjusted notification message.

[1390] Step 9:

[1391] The server provides users with additional information related to a specific condition, such as detailed documentation on a specific technology or implementation methods based on past examples. The input is text data evaluated as highly useful, and the output is the related additional information.

[1392] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1393] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1394] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1395] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1396] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1397] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1398] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1399] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1400] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1401] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1402] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1403] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1404] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1405] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1406] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1407] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1408] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1409] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1410] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1411] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1412] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1413] The following is further disclosed regarding the above embodiment.

[1414] (Claim 1)

[1415] A means for collecting user statements;

[1416] A means for converting the collected utterances into text data;

[1417] A means for extracting keywords by performing natural language processing on text data;

[1418] A means for evaluating the usefulness of text data based on the extracted keywords;

[1419] a means for notifying the user when the usefulness is rated high;

[1420] A system including:

[1421] (Claim 2)

[1422] 2. The system according to claim 1, wherein if the collected utterances are text data, they are directly input into natural language processing.

[1423] (Claim 3)

[1424] 2. The system according to claim 1, wherein if the collected utterances are voice data, they are converted into text data using voice recognition technology.

[1425] (Claim 4)

[1426] 10. The system of claim 1, further comprising means for providing additional information related to text data that has been rated as highly useful.

[1427] (Claim 5)

[1428] 2. The system according to claim 1, further comprising means for automatically collecting collected comments when the comments are made through a communication tool.

[1429] (Claim 6)

[1430] 10. The system of claim 1, further comprising means for comparing the text data with a historical database to assess its usefulness.

[1431] "Example 1"

[1432] (Claim 1)

[1433] A means for collecting user statements;

[1434] A means for converting the collected utterances into text data;

[1435] means for transmitting the text data to a central processing unit through a communication system;

[1436] A means for extracting keywords from text data using natural language analysis technology;

[1437] A means for evaluating the usefulness of text data based on the extracted keywords;

[1438] means for notifying a user based on the assessed usefulness;

[1439] a means of providing additional relevant information;

[1440] A system including:

[1441] (Claim 2)

[1442] 2. The system according to claim 1, wherein if the collected utterances are text data, they are directly input into natural language analysis.

[1443] (Claim 3)

[1444] 2. The system according to claim 1, wherein if the collected utterances are voice data, they are converted into text data using voice recognition technology.

[1445] "Application Example 1"

[1446] (Claim 1)

[1447] A means for collecting user statements;

[1448] A means for converting the collected utterances into text data;

[1449] A means for extracting keywords by performing natural language processing on text data;

[1450] A means for evaluating the usefulness of text data based on the extracted keywords;

[1451] a means for notifying the user when the usefulness is rated high;

[1452] Furthermore, a means for recognizing voice data to collect customer utterances in a physical store and displaying relevant useful information on a smart gadget;

[1453] A system including:

[1454] (Claim 2)

[1455] 2. The system according to claim 1, wherein if the collected utterances are text data, they are directly input into natural language processing.

[1456] (Claim 3)

[1457] 2. The system according to claim 1, wherein if the collected utterances are voice data, they are converted into text data using voice recognition technology.

[1458] "Example 2: Combining Emotion Engines"

[1459] (Claim 1)

[1460] A means for collecting user statements;

[1461] A means for converting the collected utterances into text data;

[1462] A means for extracting keywords by performing natural language processing on text data;

[1463] A means for evaluating the usefulness of text data based on the extracted keywords;

[1464] means for identifying a user's emotion from the text data;

[1465] a means for notifying the user when the usefulness is rated high;

[1466] a means of providing additional relevant information;

[1467] A system including:

[1468] (Claim 2)

[1469] 2. The system according to claim 1, wherein if the collected utterances are text data, they are directly input into natural language processing.

[1470] (Claim 3)

[1471] 2. The system according to claim 1, wherein if the collected utterances are voice data, they are converted into text data using voice recognition technology.

[1472] "Application example 2 when combining emotion engines"

[1473] (Claim 1)

[1474] A means for collecting user statements;

[1475] A means for converting the collected utterances into text data;

[1476] A means for extracting keywords by performing natural language processing on text data;

[1477] A means for evaluating the usefulness of text data based on the extracted keywords;

[1478] a means for notifying the user when the usefulness is rated high;

[1479] A means for analyzing a user's emotions from the text data and adjusting the notification content based on the emotions;

[1480] a means of providing additional information that is relevant under certain conditions;

[1481] A system including:

[1482] (Claim 2)

[1483] 2. The system according to claim 1, wherein if the collected utterances are text data, they are directly input into natural language processing.

[1484] (Claim 3)

[1485] 2. The system according to claim 1, wherein if the collected utterances are voice data, they are converted into text data using voice recognition technology. [Explanation of symbols]

[1486] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting user statements; A means for converting the collected utterances into text data; A means for extracting keywords by performing natural language processing on text data; A means for evaluating the usefulness of text data based on the extracted keywords; a means for notifying the user when the usefulness is rated high; A system including:

2. 2. The system according to claim 1, wherein when the collected utterances are text data, they are directly input into natural language processing.

3. 2. The system according to claim 1, wherein if the collected utterances are voice data, they are converted into text data using voice recognition technology.

4. The system of claim 1 , further comprising means for providing additional information related to text data that has been rated as highly useful.

5. 2. The system according to claim 1, further comprising means for automatically collecting collected comments when the collected comments are made through a communication tool.

6. 10. The system of claim 1, further comprising means for comparing the text data with a historical database to assess its usefulness.

Images