System

Abstract

A system is provided.SOLUTION: Means for collecting voice data and transmitting the voice data to a server in real time, means for converting the voice data received by the server into text data by an automatic voice recognition engine and analyzing the text data by a natural language processing engine, means for monitoring a progress status of a meeting based on a result of the analysis by the server and detecting an abnormal progress status, means for generating advice or a suggestion based on a result of the detection by the server and transmitting the suggestion to a terminal, and means for displaying the advice or the suggestion received by the terminal in real time, A system comprising: means for generating and transmitting a meeting playbook to a terminal using generative artificial intelligence; and means for displaying the received meeting playbook at the terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional meetings often do not proceed smoothly, and participants' comments tend to be biased. It is also difficult to grasp the content of the meeting in real time and provide appropriate advice and suggestions. Therefore, there is a need to improve the efficiency and effectiveness of meetings. In particular, in difficult meetings moderated by superiors, it is necessary to support junior employees and employees in charge of new tasks to speak up at the appropriate time and effectively advance the discussion. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system including: means for collecting voice data and transmitting it to a server in real time; means for converting the received voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine; means for monitoring the progress of the meeting based on the analysis results and detecting abnormal progress; means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal; means for the terminal to display the received advice or suggestions in real time; means for the server to analyze past meeting data and generate a meeting playbook using artificial intelligence and transmit the playbook to the terminal; and means for the terminal to display the received meeting playbook. This system makes it possible to improve the progress of meetings and the quality of discussions.

[0006] "Audio data" refers to audio information including speech made by participants during a conference.

[0007] "Server" refers to a computer system that receives, analyzes, and processes audio data.

[0008] "Terminal" refers to a computer or smart device used by a participant, and refers to a device that communicates with the server to send and receive information.

[0009] An "automatic speech recognition engine" refers to an algorithm or software that takes voice data as input and converts it into text data.

[0010] "Text data" refers to text information converted from voice data by an automatic voice recognition engine.

[0011] A "natural language processing engine" refers to algorithms or software that analyzes text data and identifies its meaning, topic, sentiment, etc.

[0012] "Analysis results" refers to information such as meaning, topic, and sentiment regarding a statement obtained by a natural language processing engine.

[0013] "Meeting progress" refers to the status of whether the meeting is proceeding according to plan and whether the agenda is being followed.

[0014] "Abnormal progress" refers to a situation where the meeting is not proceeding as planned or where a specific problem has arisen.

[0015] "Advice and suggestions" refers to specific instructions and questions generated by the server to help the meeting proceed smoothly.

[0016] A "meeting playbook" refers to guidelines and materials regarding meeting progress and discussions generated by the server based on past meeting data and related information.

[0017] "Generative artificial intelligence" refers to AI technology that generates appropriate guidelines and suggestions through data analysis and pattern recognition. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. The specific form of this system is shown below.

[0040] Overall system configuration

[0041] This system is broadly composed of three elements: a server, a terminal, and a user.

[0042] server

[0043] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and sentiment of the speech. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates advice and suggestions based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0044] Terminal

[0045] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. It also receives and displays the generated meeting playbook for the user to refer to.

[0046] User

[0047] Users are participants in a conference and make comments during the conference. The voice data of the user's speech is sent to the server via the terminal, where it is analyzed. The user can refer to the advice and suggestions displayed on the terminal to make comments at the appropriate time and to proceed with the conference.

[0048] Program processing

[0049] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and sentiment of the comments. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0050] Specific examples

[0051] Example 1: Real-time feedback

[0052] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the progress is monitored. If a topic that requires discussion has been skipped, the server generates a suggestion such as "Before moving on, please ask participant B for his opinion," and sends it to the device. The device receives this suggestion and displays it to the user, who then proceeds with the meeting by saying, "Before that, Mr. B, do you have any comments?"

[0053] Example 2: Using the Conference Playbook

[0054] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" The server sends the generated playbook to the device, which receives it and displays it on a GUI. Users can refer to the playbook to ask effective questions and keep the meeting running smoothly.

[0055] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] A user speaks during a conference, and audio data is collected by the device's microphone.

[0059] Step 2:

[0060] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[0061] Step 3:

[0062] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[0063] Step 4:

[0064] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to determine the topic, sentiment, and importance of the utterance.

[0065] Step 5:

[0066] The server monitors the progress of the meeting based on the analysis results, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[0067] Step 6:

[0068] If the server detects abnormal progress, it generates appropriate advice or suggestions, such as "You might want to seek further feedback on this topic."

[0069] Step 7:

[0070] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[0071] Step 8:

[0072] The advice and suggestions received by the device are displayed in a format that provides a good user experience (UX), presented through a GUI (Graphical User Interface).

[0073] Step 9:

[0074] Users can check the advice and suggestions displayed on their device screen and proceed with the meeting based on them, for example, by asking specific participants for their opinions or changing the focus of the discussion.

[0075] Step 10:

[0076] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[0077] Step 11:

[0078] The server generates a meeting playbook and sends it to the device, making it available before and during the meeting.

[0079] Step 12:

[0080] The terminal displays the received meeting playbook, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[0081] The process flow of this system will help meetings run smoothly and improve the quality of discussions, allowing all participants to participate effectively, greatly increasing meeting efficiency.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Improving the progress of meetings and the quality of discussions is a crucial issue in modern organizations. However, meetings often do not proceed as planned, discussions go off track, and some participants find it difficult to speak up. Furthermore, there is a lack of methods to effectively utilize past meeting data to conduct meetings effectively. To solve these issues, real-time progress management and appropriate feedback to participants are required.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the received voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the conference based on the analysis results and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the received advice or suggestions in real time, means for the server to analyze past conference data and generate a conference playbook using artificial intelligence and transmit it to the terminal, and means for the terminal to display the received conference playbook. This makes it possible to manage the progress of the conference in real time and provide appropriate advice immediately when an abnormal situation is detected. Furthermore, past conference data can be used to generate effective playbooks, improving the quality and efficiency of conferences.

[0087] "Voice data" refers to data for recording and transmitting user speech and voice information in digital format.

[0088] "Server" means a computer system that receives, analyzes, monitors, generates, and transmits audio data.

[0089] A "terminal" is a computer or smart device used by a user, and is a device for collecting voice data and displaying advice and suggestions from the server.

[0090] A "user" is a participant in a conference, who speaks and receives advice and suggestions from the server.

[0091] An "automatic speech recognition engine" is a software component for converting voice data into text data.

[0092] A "natural language processing engine" is a software component that analyzes text data and identifies meaning and sentiment.

[0093] "Monitoring" is the act or process of watching the progress of a meeting in real time and detecting anomalies.

[0094] An "abnormal progress" is when a meeting deviates from the planned course of action, certain topics are skipped, or necessary discussions are not held.

[0095] "Advice" refers to instructions or suggestions provided to facilitate the smooth running of a meeting.

[0096] A "proposal" is a specific action or instruction to solve a problem.

[0097] A "meeting playbook" is a document generated based on past meeting data that includes materials and sample questions for conducting effective meetings.

[0098] "Real-time" refers to processing or communication that occurs nearly immediately or without delay.

[0099] "Generative AI" is an artificial intelligence technology that analyzes large amounts of data, learns patterns and knowledge, and generates new information and suggestions.

[0100] System configuration

[0101] The conference support system of the present invention is composed of three main elements: a server, a terminal, and a user.

[0102] server

[0103] The server is responsible for receiving, analyzing, and monitoring the audio data, and generating advice and suggestions. Specifically, the process proceeds as follows:

[0104] Receiving and converting audio data

[0105] The server receives voice data from the device, converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API), and sends the text data to a natural language processing engine (e.g., OpenAI's GPT-3) for analysis.

[0106] Text data analysis

[0107] The server then analyzes the converted text data using a natural language processing engine to identify the topic and sentiment of the comments, and monitors the progress of the meeting in real time based on the analysis results to detect any anomalies.

[0108] Generating advice and suggestions

[0109] If the server detects an abnormality in the progress, it generates appropriate advice or suggestions. For example, it may generate a specific suggestion such as, "Please ask participant B for his / her opinion before moving on to the next agenda item." This is generated using a generative AI model (e.g., GPT-3).

[0110] Generate a conference playbook

[0111] The server analyzes past meeting data and generates a meeting playbook to support effective progress. The playbook is a document that learns and summarizes past patterns and effective questions.

[0112] Terminal

[0113] The device collects user utterances, transmits the audio data to the server, and displays advice, suggestions, and meeting playbooks received from the server in real time.

[0114] Collection and transmission of voice data

[0115] The device collects the user's speech using a microphone and transmits it to the server in real time using WebSocket or HTTP POST requests.

[0116] View advice, suggestions and playbooks

[0117] Advice and suggestions received from the server are displayed in real time on the terminal, possibly as a pop-up window or in a notification area. The meeting playbook sent from the server is also displayed on the GUI for the user to refer to in real time.

[0118] User

[0119] Users are participants in the conference and make comments. Their comments are collected by their devices and sent to the server. The progress of the conference is adjusted based on advice and suggestions displayed on the devices.

[0120] User comments and reviews

[0121] The user makes a statement and reviews the advice and suggestions displayed on the device, which determines what to say next and how to proceed.

[0122] Specific examples

[0123] Real-time feedback

[0124] 1. A user says, "Let's move on to the next item on the agenda."

[0125] 2. The device collects the voice data and sends it to the server.

[0126] 3. The server converts the voice data into text data using the Google Speech-to-Text API.

[0127] 4. The server analyzes the text data using OpenAI's GPT-3 and monitors progress.

[0128] 5. The server determines that a topic requiring discussion has been skipped and generates an advice.

[0129] 6. The server sends the advice to the terminal: "Please ask participant B for his opinion before moving on."

[0130] 7. The device receives the advice and displays it to the user in a pop-up notification.

[0131] 8. The user says, "Before that, Mr. B, do you have any comments?"

[0132] Using the Conference Playbook

[0133] 1. The server analyzes past meeting data and generates a meeting playbook containing effective questions.

[0134] 2. The server sends the generated conference playbook to the terminal.

[0135] 3. The terminal receives the conference playbook and displays it on the GUI.

[0136] 4. The user consults the meeting playbook and schedules the next question to be asked.

[0137] 5. Users effectively facilitate meetings by asking the question, "What's the progress on risk management?"

[0138] Prompt Sentence Examples

[0139] "Generate a reminder to avoid skipping topics that need discussion after saying let's move on to the next agenda item."

[0140] "Generate a playbook with useful questions for project management meetings."

[0141] In this way, the system of the present invention can support the progress of meetings and improve the quality of discussions by providing real-time feedback and utilizing past data.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] System program processing flow

[0144] Server Processing Steps

[0145] Step 1: Receiving audio data

[0146] Step 2: Convert the audio data

[0147] Step 3: Analyzing the text data

[0148] Step 4: Monitor the progress of the meeting

[0149] Step 5: Generate advice and suggestions

[0150] Step 6: Submit your advice and suggestions

[0151] Step 7: Generate the conference playbook

[0152] Terminal processing steps

[0153] Step 1: Collecting audio data

[0154] Step 2: Sending audio data

[0155] Step 3: Receiving advice and suggestions

[0156] Step 4: Viewing Advice and Suggestions

[0157] Step 5: Receive the conference playbook

[0158] Step 6: View the conference playbook

[0159] User processing steps

[0160] Step 1: Speak up

[0161] Step 2: Review advice and suggestions

[0162] Step 3: Refer to the conference playbook

[0163] Detailed explanation of the processing steps

[0164] Server Processing Steps

[0165] Step 1: Receiving audio data

[0166] The server receives audio data from the device. The input is the audio data sent from the device, and the output is the received audio data. This audio data is received via WebSocket or HTTP POST request.

[0167] Step 2: Convert the audio data

[0168] The server converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API). The input is the received voice data, and the output is the converted text data. Converting the voice data into text data prepares it for natural language processing.

[0169] Step 3: Analyzing the text data

[0170] The server analyzes the converted text data using a natural language processing engine (e.g., OpenAI's GPT-3). The input is text data, and the output is the analysis results. The analysis results include the topic and sentiment of the comments. This analysis allows for a concrete understanding of the progress of the meeting.

[0171] Step 4: Monitor the progress of the meeting

[0172] The server monitors the progress of the conference based on the results of analyzing the text data. The input is the analysis results, and the output is monitoring information. This information is used to detect abnormal progress (such as skipping topics or delays in progress).

[0173] Step 5: Generate advice and suggestions

[0174] If the server detects abnormal progress, it generates advice or suggestions. The input is monitoring information, and the output is the generated advice or suggestions. A generative AI model (e.g., GPT-3) is used to suggest specific actions. For example, it generates suggestions such as, "Please ask participant B for his opinion before moving on to the next agenda item."

[0175] Step 6: Submit your advice and suggestions

[0176] The server sends generated advice and suggestions to the device. The input is the generated advice and suggestions, and the output is the sent advice and suggestions. The low-latency WebSocket protocol is used to notify the device in real time.

[0177] Step 7: Generate the conference playbook

[0178] The server analyzes past meeting data and generates a meeting playbook. The input is past meeting data, and the output is the generated meeting playbook. A generative AI model is used to learn past patterns and effective example questions, and generate a specific playbook.

[0179] Terminal processing steps

[0180] Step 1: Collecting audio data

[0181] The device collects the user's speech from a microphone. The input is the user's speech, and the output is the collected voice data. This voice data is sent to the server in real time.

[0182] Step 2: Sending audio data

[0183] The device sends collected audio data to the server. The input is the collected audio data, and the output is the audio data sent to the server. Data is sent with low latency using WebSocket or HTTP POST requests.

[0184] Step 3: Receiving advice and suggestions

[0185] The device receives advice and suggestions sent from the server. The input is the advice and suggestions sent from the server, and the output is the received advice and suggestions. The advice and suggestions are received in real time and reflected in the next action.

[0186] Step 4: Viewing Advice and Suggestions

[0187] The advice and suggestions received by the device are displayed in real time on the GUI. The input is the received advice and suggestions, and the output is the displayed advice and suggestions. The user is notified using a pop-up window or notification area.

[0188] Step 5: Receive the conference playbook

[0189] The terminal receives the meeting playbook sent from the server. The input is the meeting playbook sent from the server, and the output is the received meeting playbook. It is received in real time and used during the meeting.

[0190] Step 6: View the conference playbook

[0191] The terminal displays the received conference playbook on the GUI. The input is the received conference playbook, and the output is the displayed conference playbook, allowing the user to refer to it immediately.

[0192] User processing steps

[0193] Step 1: Speak up

[0194] Users make comments during a conference. The input is the progress of the conference, and the output is the user's comments. These comments are collected by the terminal and sent to the server.

[0195] Step 2: Review advice and suggestions

[0196] The user reviews the advice and suggestions displayed on the device. The input is the displayed advice and suggestions, and the output is the user's review. The next action is determined based on the review result.

[0197] Step 3: Refer to the conference playbook

[0198] The user refers to the meeting playbook received from the server. The input is the displayed meeting playbook, and the output is the next action based on the reference. The playbook is used to adjust the content and timing of remarks.

[0199] In this way, the elements of the server, terminals, and users work together to realize a system that improves the progress of meetings and the quality of discussions.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Conventional meeting support systems lack effective means for improving meeting progress and the quality of discussions. In particular, in production meetings in factories and other situations where real-time feedback is required, efficient and prompt progress is essential, but current systems are insufficient. Furthermore, because they are unable to perform sentiment analysis during ongoing meetings, it is difficult to accurately detect problems that arise during meetings and provide advice and suggestions for improvement. Therefore, a new system is needed to improve the efficiency and quality of meetings.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the meeting based on the results and detecting abnormal progress, means for generating advice and suggestions and transmitting them to the terminal, means for identifying emotions and generating appropriate advice and suggestions based thereon and transmitting them to the terminal, means for generating a meeting playbook and transmitting it to the terminal, and means for displaying the advice and suggestions based on the emotions received by the terminal in real time. This makes it possible to provide appropriate feedback based on emotions in production meetings and other meetings and effectively support the progress of the meeting.

[0205] "Audio data" refers to data in which the sound of a meeting, conversation, etc. is recorded in digital format.

[0206] "Server" is a computer system for collecting, analyzing, monitoring, and generating suggestions from speech data.

[0207] An "automatic speech recognition engine" is a technology that converts voice data into text data.

[0208] A "natural language processing engine" is a technology for analyzing text data and understanding its content.

[0209] "Emotion identification" is a technology that determines the emotion of a speaker based on text data.

[0210] "Real-time" refers to processing occurring immediately without delay.

[0211] "Progress monitoring" is a technique for constantly monitoring and recording the progress of meetings and work.

[0212] "Anomaly detection" refers to discovering when progress is not normal.

[0213] "Advice and suggestion generation" is a technology that generates advice and suggestions to support the progress of meetings and work based on the analysis results.

[0214] A "meeting playbook" is a guide document generated by analyzing past meeting data to support the progress of the next meeting.

[0215] A "terminal" is a device that receives advice, suggestions, and conference playbooks sent from the server and displays them to the user.

[0216] The system of the present invention is characterized by collecting voice data in real time and analyzing it on a server in order to improve the progress of meetings and the quality of discussions. Furthermore, advice and suggestions are generated based on the analysis results, and are sent to and displayed on terminals, thereby supporting efficient progress of meetings.

[0217] Overall system configuration

[0218] This system is broadly composed of three elements: a server, a terminal, and a user.

[0219] server

[0220] The server collects, analyzes, monitors, and generates suggestions from audio data. It uses the following hardware and software:

[0221] Hardware: A server computer equipped with a high-performance processor

[0222] Software: Google Cloud Speech-to-Text API, automatic speech recognition engine, TextBlob, natural language processing engine

[0223] The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed using a natural language processing engine to identify the topic and sentiment of the speech. This allows the progress of the meeting to be monitored and abnormal progress to be detected. For example, when a speech such as "Let's move on to the next agenda item" is heard using text analysis, the system checks based on this content to see if necessary topics have been skipped. If necessary, it generates a suggestion such as "Please ask participants for their opinions before moving on."

[0224] Terminal

[0225] A terminal is a computer or smart device used by a user that has the following functions:

[0226] Audio data collection: Use a microphone to collect meeting audio

[0227] Data transmission: Collected voice data is sent to the server in real time.

[0228] Data display: Real-time display of advice, suggestions, and meeting playbooks sent from the server

[0229] The terminal receives suggestions from the server and displays them to the user, for example, "Please review your risk management before proceeding to the next step" based on the meeting playbook.

[0230] User

[0231] Users are participants in a conference and make comments during the conference. The voice data spoken by the user is sent to the server via the terminal. The results of analysis by the server are displayed on the terminal, and the user can use this information to proceed with the conference. For example, by saying, "Before we begin, do you have any comments?", the ongoing discussion can be stimulated.

[0232] Specific examples

[0233] When a user says, "There is a delay in the next production line," the voice data is sent to the server via the device. The server converts the voice data into text data using an automatic speech recognition engine and analyzes it using TextBlob. Based on the analysis results, the server generates a suggestion saying, "There appears to be a delay. Let's identify the specific problem," and sends it to the device. The device receives this suggestion and displays it to the user.

[0234] Prompt Sentence Examples

[0235] Input: "The next production line is experiencing delays."

[0236] output:

[0237] Voice recognition result: There is a delay in the next production line

[0238] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[0239] Tip: It looks like you're having a problem. Let's find out what the problem is.

[0240] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0241] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0242] Step 1:

[0243] This system collects voice data spoken by users during a conference. The input is the voice spoken by users during a conference. The device collects this voice using a built-in microphone or an external microphone and records it as digital voice data. The output is the collected digital voice data.

[0244] Step 2:

[0245] The collected voice data is sent to the server in real time. The input is the digital voice data collected in step 1. The device sends this voice data to the server via the network. The output is the voice data received by the server.

[0246] Step 3:

[0247] The server converts the received voice data into text data using an automatic speech recognition engine. The input is the voice data received by the server. The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The output is the converted text data.

[0248] Step 4:

[0249] The server analyzes the text data using a natural language processing engine to identify the topic and sentiment of the utterance. The input is the text data generated in step 3. The server analyzes the text data using TextBlob to identify the topic and sentiment. Specifically, it extracts important keywords from the text and performs sentiment analysis. The output is the topic and sentiment data as the analysis results.

[0250] Step 5:

[0251] The server monitors the progress of the meeting based on the analysis results and detects abnormal progress. The input is the topic and emotion data obtained in step 4. The server monitors the progress in real time and detects abnormal situations, such as when an important topic in the meeting is skipped. The output is the detection result when an abnormality is detected.

[0252] Step 6:

[0253] The server generates advice and suggestions based on the detection results and sends them to the terminal. The input is the anomaly detection result from step 5. The server generates advice and suggestions for the next step based on the analysis results and detection results and sends these advice and suggestions to the terminal. The output is the generated advice and suggestions.

[0254] Step 7:

[0255] The device displays the received advice or suggestion to the user in real time. The input is the advice or suggestion generated in step 6. The device displays this advice or suggestion on the screen and notifies the user. The output is the advice or suggestion that the user can see on the screen.

[0256] Step 8:

[0257] The server analyzes past meeting data and uses generative AI to generate a meeting playbook, which is then sent to the device. The input is the previously accumulated meeting data. The server uses a generative AI model to learn past patterns and effective questions, and generates the meeting playbook. The output is the generated meeting playbook.

[0258] Step 9:

[0259] The terminal displays the received conference playbook. The input is the conference playbook generated in step 8. The terminal displays the generated playbook on the screen so that the user can refer to it. The output is a conference playbook that the user can check.

[0260] Examples of prompt statements

[0261] Input: "The next production line is experiencing delays."

[0262] output:

[0263] Voice recognition result: There is a delay in the next production line

[0264] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[0265] Tip: It looks like you're having a problem. Let's find out what the problem is.

[0266] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0267] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, by combining it with an emotion engine that recognizes the user's emotions, the progress of meetings can be made more effective. Specific examples of this system are shown below.

[0268] Overall system configuration

[0269] This system is broadly composed of three elements: the server, the terminal, and the user. The server also has an emotion engine built in.

[0270] server

[0271] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. It then uses an emotion engine to recognize the user's emotions from the analysis results. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates appropriate advice and suggestions that take emotions into account based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0272] Terminal

[0273] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. The terminal also receives and displays the generated meeting playbook and emotion-related feedback for the user to refer to.

[0274] User

[0275] Users are participants in a conference and speak during the conference. The voice data they speak is sent to the server via their device, where it is analyzed. They use the advice, suggestions, and emotion-related feedback displayed on their device to speak at the appropriate time and guide the conference.

[0276] Program processing

[0277] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. Finally, it uses an emotion engine to recognize the user's emotion from the analysis results. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time, and emotional feedback is also provided. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0278] Specific examples

[0279] Example 1: Real-time feedback

[0280] The user says, "Let's move on to the next item on the agenda." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the emotion engine recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for Participant B's opinion," and send feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who can then use them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[0281] Example 2: Using the Conference Playbook

[0282] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook that includes the following questions: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also uses an emotion engine to analyze the emotional tendencies of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure the meeting proceeds smoothly.

[0283] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0284] The processing flow will be explained below.

[0285] Step 1:

[0286] A user speaks during a conference, and audio data is collected by the device's microphone.

[0287] Step 2:

[0288] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[0289] Step 3:

[0290] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[0291] Step 4:

[0292] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to identify the topic and importance of the comment.

[0293] Step 5:

[0294] The server passes the analysis results obtained by the NLP engine to the emotion engine, which recognizes the speaker's emotions. The emotion engine evaluates the type of emotion (e.g., joy, anger, anxiety, etc.) and intensity of the emotion contained in the speech.

[0295] Step 6:

[0296] The server monitors the progress of the meeting based on the analysis results and emotion data, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[0297] Step 7:

[0298] The server generates appropriate advice and suggestions based on the abnormal progress and emotion data, such as "You might want to ask for more opinions on this topic" or "Participant C is showing anxiety, so ask for their opinion."

[0299] Step 8:

[0300] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[0301] Step 9:

[0302] The device displays the received advice, suggestions, and emotion-related feedback to the user through a GUI (Graphical User Interface).

[0303] Step 10:

[0304] Users can view advice, suggestions, and emotional feedback displayed on their device screen and use it to guide the meeting, for example by asking specific participants for their opinions or changing the focus of the discussion.

[0305] Step 11:

[0306] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[0307] Step 12:

[0308] The server sends the generated meeting playbook and the results of sentiment analysis to the device, making them available before and during the meeting.

[0309] Step 13:

[0310] The device displays the received meeting playbook and sentiment analysis results, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[0311] Specific examples

[0312] Example 1: Real-time feedback

[0313] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine. The emotion engine then recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for participant B's opinion," and send it to the device in real time. The device displays this, and the user can respond to the suggestion by saying, "Before that, Mr. B, do you have any comments?", thereby ensuring the smooth progress of the meeting.

[0314] Example 2: Using the Conference Playbook

[0315] The server analyzes past meeting data and learns patterns using generative artificial intelligence. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It then uses an emotion engine to analyze the emotional trends of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. The user can use this as a reference to ask effective questions and keep the meeting running smoothly.

[0316] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0317] Example 2

[0318] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] Conventional meeting support systems did not take into account the emotions of participants, which resulted in problems such as meetings stalling or making it difficult for some participants to speak up. Furthermore, they lacked the functionality to monitor meeting progress in real time and provide appropriate advice and suggestions, making it difficult to facilitate effective discussions. Furthermore, there were insufficient means to effectively utilize past meeting data, which often led to repeated discussions of the same issues.

[0320] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0321] In this invention, the server includes: means for receiving voice data and converting it into text data using an automatic speech recognition engine; means for analyzing it using a natural language processing engine; means for identifying the user's emotions based on the analysis results, monitoring the progress of the meeting, and detecting abnormal progress; means for generating advice and suggestions based on the detection results and the identified emotions and transmitting the suggestions to the terminal; and means for analyzing past meeting data, generating a meeting playbook using a generative AI model, and transmitting the playbook to the terminal. This enables real-time support for the progress of meetings and the provision of advice and suggestions that take participants' emotions into consideration. Furthermore, past meeting data can be utilized to generate and display a meeting playbook to guide effective discussions.

[0322] "Audio data" refers to recorded information of words or remarks made by users during a conference.

[0323] "Server" refers to a computer system that receives, analyzes, identifies emotions, monitors, generates advice, and generates meeting playbooks for voice data.

[0324] "Terminal" refers to a computer or smart device used by a user that collects voice data and displays advice, suggestions, and meeting playbooks sent from the server.

[0325] An "automatic speech recognition engine" refers to software technology for analyzing voice data and converting it into text data.

[0326] "Text data" refers to character information converted from voice data by an automatic voice recognition engine.

[0327] A "natural language processing engine" refers to a technology that analyzes text data, understands its content and context, and extracts information.

[0328] "Emotion engine" refers to software technology that identifies a user's emotions based on text data.

[0329] "Monitoring" refers to the process of observing the progress of a meeting in real time and detecting any anomalies or problems.

[0330] "Advice" refers to recommendations and suggestions generated by the server to improve the progress of the meeting and the quality of the discussion.

[0331] "Suggestion" refers to a specific solution or instruction for action provided by the server for a particular situation or problem.

[0332] A "generative AI model" refers to an artificial intelligence model that learns from past data and makes predictions and generates based on new data.

[0333] "Meeting playbooks" refer to templates and guidelines for conducting effective discussions, created by a generative AI model based on past meeting data.

[0334] "Real-time display" refers to the immediate display of received advice or suggestions on the device.

[0335] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, this system makes meetings more effective by combining it with an emotion engine that recognizes the user's emotions. The specific form of this system is shown below.

[0336] Overall system configuration

[0337] This system is roughly divided into three components: a server, a terminal, and a user. The server has an emotion engine built in.

[0338] Server Features

[0339] The server collects, analyzes, monitors, and generates suggestions for voice data. Specifically, the server receives voice data and converts it into text using an automatic speech recognition engine such as Google Cloud Speech-to-Text. It then analyzes the text data using a natural language processing engine such as GPT-4 to identify the topic and sentiment of the speech. It then uses an emotion engine such as IBM Watson Tone Analyzer to recognize the user's sentiment from the analysis results. This allows the server to monitor the progress of the meeting and detect any abnormalities.

[0340] If an anomaly is detected, the server generates advice or suggestions using a generative AI model (e.g., GPT-3) and sends them to the device in real time. The server also analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). This playbook provides templates to help facilitate effective discussions.

[0341] Device Features

[0342] The terminal is a computer or smart device used by the user that collects voice data and sends it to the server. The terminal also displays real-time advice and suggestions sent from the server and provides them to the user. It also receives and displays the generated meeting playbook and sentiment-related feedback. This can be achieved through a custom GUI in a web browser or a plugin for the meeting software (e.g., Zoom, Microsoft Teams).

[0343] User Roles

[0344] Users are participants in a meeting and generate voice data by speaking. This voice data is sent to the server via their devices. Users refer to the advice and suggestions sent from the server to guide the meeting appropriately. Users can also refer to the generated meeting playbook to receive support for effective question and answer sessions and discussions.

[0345] Specific examples

[0346] Example 1: Real-time feedback

[0347] User A says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using Google Cloud Speech-to-Text. Next, it analyzes the text data using the natural language processing engine GPT-4 and recognizes the speaker's emotions using IBM Watson Tone Analyzer. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server generates a suggestion such as "Before that, please ask for participant B's opinion," and sends feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who then uses them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[0348] Prompt Sentence Examples

[0349] "User A wants to move on to the next agenda item, but seems impatient. Suggest that they listen to Participant B's opinion."

[0350] Example 2: Using the Conference Playbook

[0351] The server analyzes past project management meeting data and learns patterns using a generative AI model (e.g., BERT). For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also analyzes the emotional trends of participants in past meetings using IBM Watson Tone Analyzer. The server then sends the generated playbook and advice based on the emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure smooth progress in meetings.

[0352] Prompt Sentence Examples

[0353] "This is a meeting playbook generated from past meeting data. Please ask questions and make suggestions to elicit opinions from your team members."

[0354] This system combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0355] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0356] Step 1: Collecting audio data

[0357] The device collects the voice data of the user during the conference. The voice data is recorded through the device's microphone. The input is the user's speech, and the output is the recorded voice data. Specifically, the conference application captures the voice data in real time and prepares it to be sent to the server.

[0358] Step 2: Sending the audio data to the server

[0359] The device sends the collected voice data to the server. The input is the recorded voice data, and the output is the voice data sent to the server. Specifically, the device's communication module is used to encrypt the voice data and send it to the server via the network.

[0360] Step 3: Converting audio data to text

[0361] The server converts the received voice data into text data using an automatic speech recognition engine such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data. Specifically, the automatic speech recognition engine extracts words and sentences from the voice and outputs them as structured text data.

[0362] Step 4: Analyzing the text data

[0363] The server analyzes the text data using a natural language processing engine such as GPT-4. The input is the text data, and the output is the analyzed text data and its topic and context information. Specifically, the natural language processing engine interprets the content of the text data, extracts important keywords and phrases, and classifies them according to topic and context.

[0364] Step 5: Identify the emotion

[0365] The server uses an emotion engine such as IBM Watson Tone Analyzer to identify the user's emotion from the analyzed text data. The input is the analyzed text data, and the output is the identified emotion information. Specifically, the emotion engine analyzes the wording and structure in the text to identify the user's emotional state (e.g., joy, sadness, anger).

[0366] Step 6: Monitor the progress of the meeting

[0367] The server monitors the progress of the meeting in real time based on the identified emotions and text data. The input is emotion information and analyzed text data, and the output is the current progress of the meeting and the detection results of abnormal progress. Specifically, the server tracks this data and analyzes the frequency and content of comments, participants' reactions, etc.

[0368] Step 7: Generate advice and suggestions

[0369] The server generates advice or suggestions using a generative AI model (e.g., GPT-3) based on the detection results and identified emotions. The input is the abnormal progress detection results and emotion information, and the output is text data of the advice or suggestions. Specifically, the server forms specific suggestions such as "In this situation, please say this next" based on the generative AI model.

[0370] Step 8: Submit and view tips and suggestions

[0371] The server sends the generated advice or suggestion to the terminal, which displays it in real time. The input is text data of the advice or suggestion, and the output is the advice or suggestion displayed on the terminal. Specifically, it is displayed to the user as a pop-up window or a notification message using the terminal's display device.

[0372] Step 9: Analyze historical data and generate meeting playbooks

[0373] The server analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). The input is past meeting data, and the output is the generated meeting playbook. Specifically, the generative AI model learns from the data, extracts patterns of successful discussions and effective questions, and compiles them into templates.

[0374] Step 10: Submit and view the meeting playbook

[0375] The server sends the generated conference playbook to the terminal, which receives and displays it. The input is the generated conference playbook, and the output is the conference playbook displayed on the terminal. Specifically, it is provided to the user as a template or guideline displayed on the GUI.

[0376] (Application example 2)

[0377] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0378] Current food delivery services lack a system that can quickly respond to the difficulties and emotional changes that delivery workers face in the field. This can easily lead to delivery delays and reduced customer satisfaction. To provide an environment where delivery workers can perform their work efficiently and with peace of mind, a system is needed that monitors the progress of deliveries in real time and provides appropriate support and suggestions.

[0379] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0380] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data received by the server into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of delivery based on the results of the analysis by the server and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the advice or suggestions received in real time, means for the server to analyze past delivery data and generate a delivery playbook using generative artificial intelligence and transmit it to the terminal, and means for the terminal to display the delivery playbook received. This enables delivery personnel to respond quickly and appropriately at the scene.

[0381] "Voice data" is information collected in digital form from the voices of delivery personnel.

[0382] "Real-time" means that data is processed immediately, without delay.

[0383] The "server" is a central system that receives and analyzes voice data and sends appropriate suggestions and feedback to the device.

[0384] An "automatic speech recognition engine" is a technology for converting voice data into text data.

[0385] "Text data" is character information converted by a voice recognition engine.

[0386] A "natural language processing engine" is a technology for analyzing text data and identifying meaning and emotion.

[0387] "Progress" refers to the current status in the delivery process.

[0388] "Monitoring" means watching progress in real time.

[0389] "Abnormal" refers to a situation where delivery is not proceeding as planned.

[0390] "Advice" is a suggestion of specific action to the delivery person.

[0391] A "suggestion" is advice for the delivery person to improve the current situation.

[0392] A "terminal" is a receiving device carried by a delivery person.

[0393] "Historical delivery data" is a collection of information about deliveries that have been made previously.

[0394] "Generative artificial intelligence" is a technology for generating new insights and suggestions from data.

[0395] The "Delivery Playbook" is a guidebook created based on past delivery data to support efficient delivery.

[0396] "Display" means providing information visually on a terminal.

[0397] This invention relates to a system that supports food delivery operations. In particular, it is a system for realizing a smart device application that recognizes the emotions and situations of delivery personnel in real time and provides appropriate advice and suggestions. This system is broadly composed of three elements: a server, a terminal, and a delivery personnel. The specific form of the system is shown below.

[0398] Overall system configuration

[0399] This system sends the delivery person's voice data to a server in real time, where it is converted into text data using an automatic speech recognition engine. The voice data is then analyzed using a natural language processing engine to monitor the progress of the delivery. If an abnormality is detected, the server generates appropriate advice or suggestions and sends them to the terminal. The terminal displays these advice and suggestions in real time, helping the delivery person quickly choose their next action. The server also analyzes past delivery data and uses generative artificial intelligence to generate a delivery playbook, which is sent to the terminal. The delivery playbook serves as a guideline for efficiently carrying out delivery work.

[0400] Hardware and software used

[0401] Hardware

[0402] Smartphone: A device carried by delivery personnel that collects voice data and displays advice and suggestions.

[0403] Server: A central system that analyzes data, generates proposals, and analyzes historical data.

[0404] software

[0405] Automatic speech recognition engine: Converts voice data into text data (e.g., Google Cloud Speech-to-Text).

[0406] Natural language processing engines: Analyze text data and identify meaning and sentiment (e.g., spaCy, NLTK).

[0407] Emotion Recognition AI: Recognize emotions from text data (e.g., Microsoft Azure Emotion API).

[0408] Generative AI: Generate new insights and suggestions from data (e.g., OpenAI GPT-4).

[0409] Program processing

[0410] Server processing

[0411] The server receives voice data sent by the delivery person. It then uses an automatic speech recognition engine to convert the voice data into text data. The converted text data is analyzed by a natural language processing engine to identify the content and emotion of the speech. Emotion recognition AI is used to recognize the emotions of the delivery person and monitor the delivery progress. If abnormal progress is detected, the server uses generation AI to generate appropriate advice or suggestions and send them to the device.

[0412] Specific example explanation

[0413] Example 1: Real-time support

[0414] When a delivery person says, "I'm going to be late," this voice data is sent from the smartphone to a server. The server receives the voice data and converts it into text using Google Cloud Speech-to-Text. The text data is analyzed using spaCy, and the keyword "I'm going to be late" is extracted. When analyzed using the Microsoft Azure Emotion API, it is identified that the delivery person is feeling anxious. The suggestion generated using OpenAI GPT-4 is the message, "Please check if there is a detour nearby. Also, if congestion persists, we recommend that you report the delay to customers and take a break." This message is immediately displayed on the delivery person's smartphone. Below is an example of a prompt.

[0415] Example prompt: What is the next action the delivery person should take if "It looks like it's going to be late" and "The roads are busy"? Generate a feedback statement.

[0416] In this way, delivery personnel can respond quickly and appropriately on site.

[0417] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0418] Step 1:

[0419] During delivery, the user inputs voice data into the smartphone, for example, "I'm going to be late, the roads are congested." This voice data is collected and sent from the smartphone to the server in real time.

[0420] Input: Delivery person's voice data

[0421] Output: Audio data sent to the server

[0422] Step 2:

[0423] The server converts the received voice data into text data using an automatic speech recognition engine (Google Cloud Speech-to-Text). Through this process, the voice data is converted into text information.

[0424] Input: Audio data

[0425] Output: Text data

[0426] Step 3:

[0427] The server analyzes the obtained text data using a natural language processing engine (spaCy or NLTK). Keywords in the text data (e.g., "I'm going to be late" or "The roads are congested") are extracted, and the topic and content of the utterance are identified.

[0428] Input: Text data

[0429] Output: Parsed topic and content information

[0430] Step 4:

[0431] The server uses emotion recognition AI (Microsoft Azure Emotion API) to recognize the emotions of the delivery person from the analysis results. Emotion recognition determines whether the delivery person is feeling anxious or stressed from the text data.

[0432] Input: Parsed topic and content information

[0433] Output: Recognized emotion data

[0434] Step 5:

[0435] The server monitors the delivery progress based on the analysis results and sentiment data, and if an abnormal progress (e.g., risk of delay) is detected, it proceeds to the next step.

[0436] Input: Recognized emotion data

[0437] Output: Progress determination result

[0438] Step 6:

[0439] The server uses a generative AI model (OpenAI GPT-4) to generate appropriate advice and suggestions. The prompt sentence is set as "What is the next action the delivery person should take if the delivery person is likely to be late and the roads are congested?" and the generated suggestions are created.

[0440] Input: Progress status determination result

[0441] Output: Generated advice and suggestions

[0442] Step 7:

[0443] The server sends the generated advice and suggestions to the terminal.

[0444] Input: Generated advice and suggestions

[0445] Output: Tips and suggestions sent to the terminal

[0446] Step 8:

[0447] The device displays the received advice and suggestions in real time. The delivery person can then refer to these advice and suggestions and select the appropriate action to carry out their task. Specifically, they can check for detours according to the example sentences displayed, or report delays to the customer.

[0448] Input: Tips and suggestions sent to your device

[0449] Output: Tips and suggestions displayed on the terminal

[0450] This allows delivery personnel to respond quickly and appropriately on site.

[0451] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0452] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0453] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0454] [Second embodiment]

[0455] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0456] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0457] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0458] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0459] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0460] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0461] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0462] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0463] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0464] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0465] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0466] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0467] The present invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. The specific form of this system is shown below.

[0468] Overall system configuration

[0469] This system is broadly composed of three elements: a server, a terminal, and a user.

[0470] server

[0471] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and sentiment of the speech. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates advice and suggestions based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0472] Terminal

[0473] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. It also receives and displays the generated meeting playbook for the user to refer to.

[0474] User

[0475] Users are participants in a conference and make comments during the conference. The voice data of the user's speech is sent to the server via the terminal, where it is analyzed. The user can refer to the advice and suggestions displayed on the terminal to make comments at the appropriate time and to proceed with the conference.

[0476] Program processing

[0477] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and sentiment of the comments. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0478] Specific examples

[0479] Example 1: Real-time feedback

[0480] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the progress is monitored. If a topic that requires discussion has been skipped, the server generates a suggestion such as "Before moving on, please ask participant B for his opinion," and sends it to the device. The device receives this suggestion and displays it to the user, who then proceeds with the meeting by saying, "Before that, Mr. B, do you have any comments?"

[0481] Example 2: Using the Conference Playbook

[0482] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" The server sends the generated playbook to the device, which receives it and displays it on a GUI. Users can refer to the playbook to ask effective questions and keep the meeting running smoothly.

[0483] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0484] The processing flow will be explained below.

[0485] Step 1:

[0486] A user speaks during a conference, and audio data is collected by the device's microphone.

[0487] Step 2:

[0488] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[0489] Step 3:

[0490] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[0491] Step 4:

[0492] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to determine the topic, sentiment, and importance of the utterance.

[0493] Step 5:

[0494] The server monitors the progress of the meeting based on the analysis results, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[0495] Step 6:

[0496] If the server detects abnormal progress, it generates appropriate advice or suggestions, such as "You might want to seek further feedback on this topic."

[0497] Step 7:

[0498] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[0499] Step 8:

[0500] The advice and suggestions received by the device are displayed in a format that provides a good user experience (UX), presented through a GUI (Graphical User Interface).

[0501] Step 9:

[0502] Users can check the advice and suggestions displayed on their device screen and proceed with the meeting based on them, for example, by asking specific participants for their opinions or changing the focus of the discussion.

[0503] Step 10:

[0504] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[0505] Step 11:

[0506] The server generates a meeting playbook and sends it to the device, making it available before and during the meeting.

[0507] Step 12:

[0508] The terminal displays the received meeting playbook, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[0509] The process flow of this system will help meetings run smoothly and improve the quality of discussions, allowing all participants to participate effectively, greatly increasing meeting efficiency.

[0510] Example 1

[0511] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0512] Improving the progress of meetings and the quality of discussions is a crucial issue in modern organizations. However, meetings often do not proceed as planned, discussions go off track, and some participants find it difficult to speak up. Furthermore, there is a lack of methods to effectively utilize past meeting data to conduct meetings effectively. To solve these issues, real-time progress management and appropriate feedback to participants are required.

[0513] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0514] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the received voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the conference based on the analysis results and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the received advice or suggestions in real time, means for the server to analyze past conference data and generate a conference playbook using artificial intelligence and transmit it to the terminal, and means for the terminal to display the received conference playbook. This makes it possible to manage the progress of the conference in real time and provide appropriate advice immediately when an abnormal situation is detected. Furthermore, past conference data can be used to generate effective playbooks, improving the quality and efficiency of conferences.

[0515] "Voice data" refers to data for recording and transmitting user speech and voice information in digital format.

[0516] "Server" means a computer system that receives, analyzes, monitors, generates, and transmits audio data.

[0517] A "terminal" is a computer or smart device used by a user, and is a device for collecting voice data and displaying advice and suggestions from the server.

[0518] A "user" is a participant in a conference, who speaks and receives advice and suggestions from the server.

[0519] An "automatic speech recognition engine" is a software component for converting voice data into text data.

[0520] A "natural language processing engine" is a software component that analyzes text data and identifies meaning and sentiment.

[0521] "Monitoring" is the act or process of watching the progress of a meeting in real time and detecting anomalies.

[0522] An "abnormal progress" is when a meeting deviates from the planned course of action, certain topics are skipped, or necessary discussions are not held.

[0523] "Advice" refers to instructions or suggestions provided to facilitate the smooth running of a meeting.

[0524] A "proposal" is a specific action or instruction to solve a problem.

[0525] A "meeting playbook" is a document generated based on past meeting data that includes materials and sample questions for conducting effective meetings.

[0526] "Real-time" refers to processing or communication that occurs nearly immediately or without delay.

[0527] "Generative AI" is an artificial intelligence technology that analyzes large amounts of data, learns patterns and knowledge, and generates new information and suggestions.

[0528] System configuration

[0529] The conference support system of the present invention is composed of three main elements: a server, a terminal, and a user.

[0530] server

[0531] The server is responsible for receiving, analyzing, and monitoring the audio data, and generating advice and suggestions. Specifically, the process proceeds as follows:

[0532] Receiving and converting audio data

[0533] The server receives voice data from the device, converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API), and sends the text data to a natural language processing engine (e.g., OpenAI's GPT-3) for analysis.

[0534] Text data analysis

[0535] The server then analyzes the converted text data using a natural language processing engine to identify the topic and sentiment of the comments, and monitors the progress of the meeting in real time based on the analysis results to detect any anomalies.

[0536] Generating advice and suggestions

[0537] If the server detects an abnormality in the progress, it generates appropriate advice or suggestions. For example, it may generate a specific suggestion such as, "Please ask participant B for his / her opinion before moving on to the next agenda item." This is generated using a generative AI model (e.g., GPT-3).

[0538] Generate a conference playbook

[0539] The server analyzes past meeting data and generates a meeting playbook to support effective progress. The playbook is a document that learns and summarizes past patterns and effective questions.

[0540] Terminal

[0541] The device collects user utterances, transmits the audio data to the server, and displays advice, suggestions, and meeting playbooks received from the server in real time.

[0542] Collection and transmission of voice data

[0543] The device collects the user's speech using a microphone and transmits it to the server in real time using WebSocket or HTTP POST requests.

[0544] View advice, suggestions and playbooks

[0545] Advice and suggestions received from the server are displayed in real time on the terminal, possibly as a pop-up window or in a notification area. The meeting playbook sent from the server is also displayed on the GUI for the user to refer to in real time.

[0546] User

[0547] Users are participants in the conference and make comments. Their comments are collected by their devices and sent to the server. The progress of the conference is adjusted based on advice and suggestions displayed on the devices.

[0548] User comments and reviews

[0549] The user makes a statement and reviews the advice and suggestions displayed on the device, which determines what to say next and how to proceed.

[0550] Specific examples

[0551] Real-time feedback

[0552] 1. A user says, "Let's move on to the next item on the agenda."

[0553] 2. The device collects the voice data and sends it to the server.

[0554] 3. The server converts the voice data into text data using the Google Speech-to-Text API.

[0555] 4. The server analyzes the text data using OpenAI's GPT-3 and monitors progress.

[0556] 5. The server determines that a topic requiring discussion has been skipped and generates an advice.

[0557] 6. The server sends the advice to the terminal: "Please ask participant B for his opinion before moving on."

[0558] 7. The device receives the advice and displays it to the user in a pop-up notification.

[0559] 8. The user says, "Before that, Mr. B, do you have any comments?"

[0560] Using the Conference Playbook

[0561] 1. The server analyzes past meeting data and generates a meeting playbook containing effective questions.

[0562] 2. The server sends the generated conference playbook to the terminal.

[0563] 3. The terminal receives the conference playbook and displays it on the GUI.

[0564] 4. The user consults the meeting playbook and schedules the next question to be asked.

[0565] 5. Users effectively facilitate meetings by asking the question, "What's the progress on risk management?"

[0566] Prompt Sentence Examples

[0567] "Generate a reminder to avoid skipping topics that need discussion after saying let's move on to the next agenda item."

[0568] "Generate a playbook with useful questions for project management meetings."

[0569] In this way, the system of the present invention can support the progress of meetings and improve the quality of discussions by providing real-time feedback and utilizing past data.

[0570] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0571] System program processing flow

[0572] Server Processing Steps

[0573] Step 1: Receiving audio data

[0574] Step 2: Convert the audio data

[0575] Step 3: Analyzing the text data

[0576] Step 4: Monitor the progress of the meeting

[0577] Step 5: Generate advice and suggestions

[0578] Step 6: Submit your advice and suggestions

[0579] Step 7: Generate the conference playbook

[0580] Terminal processing steps

[0581] Step 1: Collecting audio data

[0582] Step 2: Sending audio data

[0583] Step 3: Receiving advice and suggestions

[0584] Step 4: Viewing Advice and Suggestions

[0585] Step 5: Receive the conference playbook

[0586] Step 6: View the conference playbook

[0587] User processing steps

[0588] Step 1: Speak up

[0589] Step 2: Review advice and suggestions

[0590] Step 3: Refer to the conference playbook

[0591] Detailed explanation of the processing steps

[0592] Server Processing Steps

[0593] Step 1: Receiving audio data

[0594] The server receives audio data from the device. The input is the audio data sent from the device, and the output is the received audio data. This audio data is received via WebSocket or HTTP POST request.

[0595] Step 2: Convert the audio data

[0596] The server converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API). The input is the received voice data, and the output is the converted text data. Converting the voice data into text data prepares it for natural language processing.

[0597] Step 3: Analyzing the text data

[0598] The server analyzes the converted text data using a natural language processing engine (e.g., OpenAI's GPT-3). The input is text data, and the output is the analysis results. The analysis results include the topic and sentiment of the comments. This analysis allows for a concrete understanding of the progress of the meeting.

[0599] Step 4: Monitor the progress of the meeting

[0600] The server monitors the progress of the conference based on the results of analyzing the text data. The input is the analysis results, and the output is monitoring information. This information is used to detect abnormal progress (such as skipping topics or delays in progress).

[0601] Step 5: Generate advice and suggestions

[0602] If the server detects abnormal progress, it generates advice or suggestions. The input is monitoring information, and the output is the generated advice or suggestions. A generative AI model (e.g., GPT-3) is used to suggest specific actions. For example, it generates suggestions such as, "Please ask participant B for his opinion before moving on to the next agenda item."

[0603] Step 6: Submit your advice and suggestions

[0604] The server sends generated advice and suggestions to the device. The input is the generated advice and suggestions, and the output is the sent advice and suggestions. The low-latency WebSocket protocol is used to notify the device in real time.

[0605] Step 7: Generate the conference playbook

[0606] The server analyzes past meeting data and generates a meeting playbook. The input is past meeting data, and the output is the generated meeting playbook. A generative AI model is used to learn past patterns and effective example questions, and generate a specific playbook.

[0607] Terminal processing steps

[0608] Step 1: Collecting audio data

[0609] The device collects the user's speech from a microphone. The input is the user's speech, and the output is the collected voice data. This voice data is sent to the server in real time.

[0610] Step 2: Sending audio data

[0611] The device sends collected audio data to the server. The input is the collected audio data, and the output is the audio data sent to the server. Data is sent with low latency using WebSocket or HTTP POST requests.

[0612] Step 3: Receiving advice and suggestions

[0613] The device receives advice and suggestions sent from the server. The input is the advice and suggestions sent from the server, and the output is the received advice and suggestions. The advice and suggestions are received in real time and reflected in the next action.

[0614] Step 4: Viewing Advice and Suggestions

[0615] The advice and suggestions received by the device are displayed in real time on the GUI. The input is the received advice and suggestions, and the output is the displayed advice and suggestions. The user is notified using a pop-up window or notification area.

[0616] Step 5: Receive the conference playbook

[0617] The terminal receives the meeting playbook sent from the server. The input is the meeting playbook sent from the server, and the output is the received meeting playbook. It is received in real time and used during the meeting.

[0618] Step 6: View the conference playbook

[0619] The terminal displays the received conference playbook on the GUI. The input is the received conference playbook, and the output is the displayed conference playbook, allowing the user to refer to it immediately.

[0620] User processing steps

[0621] Step 1: Speak up

[0622] Users make comments during a conference. The input is the progress of the conference, and the output is the user's comments. These comments are collected by the terminal and sent to the server.

[0623] Step 2: Review advice and suggestions

[0624] The user reviews the advice and suggestions displayed on the device. The input is the displayed advice and suggestions, and the output is the user's review. The next action is determined based on the review result.

[0625] Step 3: Refer to the conference playbook

[0626] The user refers to the meeting playbook received from the server. The input is the displayed meeting playbook, and the output is the next action based on the reference. The playbook is used to adjust the content and timing of remarks.

[0627] In this way, the elements of the server, terminals, and users work together to realize a system that improves the progress of meetings and the quality of discussions.

[0628] (Application example 1)

[0629] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0630] Conventional meeting support systems lack effective means for improving meeting progress and the quality of discussions. In particular, in production meetings in factories and other situations where real-time feedback is required, efficient and prompt progress is essential, but current systems are insufficient. Furthermore, because they are unable to perform sentiment analysis during ongoing meetings, it is difficult to accurately detect problems that arise during meetings and provide advice and suggestions for improvement. Therefore, a new system is needed to improve the efficiency and quality of meetings.

[0631] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0632] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the meeting based on the results and detecting abnormal progress, means for generating advice and suggestions and transmitting them to the terminal, means for identifying emotions and generating appropriate advice and suggestions based thereon and transmitting them to the terminal, means for generating a meeting playbook and transmitting it to the terminal, and means for displaying the advice and suggestions based on the emotions received by the terminal in real time. This makes it possible to provide appropriate feedback based on emotions in production meetings and other meetings and effectively support the progress of the meeting.

[0633] "Audio data" refers to data in which the sound of a meeting, conversation, etc. is recorded in digital format.

[0634] "Server" is a computer system for collecting, analyzing, monitoring, and generating suggestions from speech data.

[0635] An "automatic speech recognition engine" is a technology that converts voice data into text data.

[0636] A "natural language processing engine" is a technology for analyzing text data and understanding its content.

[0637] "Emotion identification" is a technology that determines the emotion of a speaker based on text data.

[0638] "Real-time" refers to processing occurring immediately without delay.

[0639] "Progress monitoring" is a technique for constantly monitoring and recording the progress of meetings and work.

[0640] "Anomaly detection" refers to discovering when progress is not normal.

[0641] "Advice and suggestion generation" is a technology that generates advice and suggestions to support the progress of meetings and work based on the analysis results.

[0642] A "meeting playbook" is a guide document generated by analyzing past meeting data to support the progress of the next meeting.

[0643] A "terminal" is a device that receives advice, suggestions, and conference playbooks sent from the server and displays them to the user.

[0644] The system of the present invention is characterized by collecting voice data in real time and analyzing it on a server in order to improve the progress of meetings and the quality of discussions. Furthermore, advice and suggestions are generated based on the analysis results, and are sent to and displayed on terminals, thereby supporting efficient progress of meetings.

[0645] Overall system configuration

[0646] This system is broadly composed of three elements: a server, a terminal, and a user.

[0647] server

[0648] The server collects, analyzes, monitors, and generates suggestions from audio data. It uses the following hardware and software:

[0649] Hardware: A server computer equipped with a high-performance processor

[0650] Software: Google Cloud Speech-to-Text API, automatic speech recognition engine, TextBlob, natural language processing engine

[0651] The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed using a natural language processing engine to identify the topic and sentiment of the speech. This allows the progress of the meeting to be monitored and abnormal progress to be detected. For example, when a speech such as "Let's move on to the next agenda item" is heard using text analysis, the system checks based on this content to see if necessary topics have been skipped. If necessary, it generates a suggestion such as "Please ask participants for their opinions before moving on."

[0652] Terminal

[0653] A terminal is a computer or smart device used by a user that has the following functions:

[0654] Audio data collection: Use a microphone to collect meeting audio

[0655] Data transmission: Collected voice data is sent to the server in real time.

[0656] Data display: Real-time display of advice, suggestions, and meeting playbooks sent from the server

[0657] The terminal receives suggestions from the server and displays them to the user, for example, "Please review your risk management before proceeding to the next step" based on the meeting playbook.

[0658] User

[0659] Users are participants in a conference and make comments during the conference. The voice data spoken by the user is sent to the server via the terminal. The results of analysis by the server are displayed on the terminal, and the user can use this information to proceed with the conference. For example, by saying, "Before we begin, do you have any comments?", the ongoing discussion can be stimulated.

[0660] Specific examples

[0661] When a user says, "There is a delay in the next production line," the voice data is sent to the server via the device. The server converts the voice data into text data using an automatic speech recognition engine and analyzes it using TextBlob. Based on the analysis results, the server generates a suggestion saying, "There appears to be a delay. Let's identify the specific problem," and sends it to the device. The device receives this suggestion and displays it to the user.

[0662] Prompt Sentence Examples

[0663] Input: "The next production line is experiencing delays."

[0664] output:

[0665] Voice recognition result: There is a delay in the next production line

[0666] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[0667] Tip: It looks like you're having a problem. Let's find out what the problem is.

[0668] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0669] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0670] Step 1:

[0671] This system collects voice data spoken by users during a conference. The input is the voice spoken by users during a conference. The device collects this voice using a built-in microphone or an external microphone and records it as digital voice data. The output is the collected digital voice data.

[0672] Step 2:

[0673] The collected voice data is sent to the server in real time. The input is the digital voice data collected in step 1. The device sends this voice data to the server via the network. The output is the voice data received by the server.

[0674] Step 3:

[0675] The server converts the received voice data into text data using an automatic speech recognition engine. The input is the voice data received by the server. The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The output is the converted text data.

[0676] Step 4:

[0677] The server analyzes the text data using a natural language processing engine to identify the topic and sentiment of the utterance. The input is the text data generated in step 3. The server analyzes the text data using TextBlob to identify the topic and sentiment. Specifically, it extracts important keywords from the text and performs sentiment analysis. The output is the topic and sentiment data as the analysis results.

[0678] Step 5:

[0679] The server monitors the progress of the meeting based on the analysis results and detects abnormal progress. The input is the topic and emotion data obtained in step 4. The server monitors the progress in real time and detects abnormal situations, such as when an important topic in the meeting is skipped. The output is the detection result when an abnormality is detected.

[0680] Step 6:

[0681] The server generates advice and suggestions based on the detection results and sends them to the terminal. The input is the anomaly detection result from step 5. The server generates advice and suggestions for the next step based on the analysis results and detection results and sends these advice and suggestions to the terminal. The output is the generated advice and suggestions.

[0682] Step 7:

[0683] The device displays the received advice or suggestion to the user in real time. The input is the advice or suggestion generated in step 6. The device displays this advice or suggestion on the screen and notifies the user. The output is the advice or suggestion that the user can see on the screen.

[0684] Step 8:

[0685] The server analyzes past meeting data and uses generative AI to generate a meeting playbook, which is then sent to the device. The input is the previously accumulated meeting data. The server uses a generative AI model to learn past patterns and effective questions, and generates the meeting playbook. The output is the generated meeting playbook.

[0686] Step 9:

[0687] The terminal displays the received conference playbook. The input is the conference playbook generated in step 8. The terminal displays the generated playbook on the screen so that the user can refer to it. The output is a conference playbook that the user can check.

[0688] Examples of prompt statements

[0689] Input: "The next production line is experiencing delays."

[0690] output:

[0691] Voice recognition result: There is a delay in the next production line

[0692] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[0693] Tip: It looks like you're having a problem. Let's find out what the problem is.

[0694] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0695] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, by combining it with an emotion engine that recognizes the user's emotions, the progress of meetings can be made more effective. Specific examples of this system are shown below.

[0696] Overall system configuration

[0697] This system is broadly composed of three elements: the server, the terminal, and the user. The server also has an emotion engine built in.

[0698] server

[0699] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. It then uses an emotion engine to recognize the user's emotions from the analysis results. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates appropriate advice and suggestions that take emotions into account based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0700] Terminal

[0701] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. The terminal also receives and displays the generated meeting playbook and emotion-related feedback for the user to refer to.

[0702] User

[0703] Users are participants in a conference and speak during the conference. The voice data they speak is sent to the server via their device, where it is analyzed. They use the advice, suggestions, and emotion-related feedback displayed on their device to speak at the appropriate time and guide the conference.

[0704] Program processing

[0705] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. Finally, it uses an emotion engine to recognize the user's emotion from the analysis results. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time, and emotional feedback is also provided. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0706] Specific examples

[0707] Example 1: Real-time feedback

[0708] The user says, "Let's move on to the next item on the agenda." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the emotion engine recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for Participant B's opinion," and send feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who can then use them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[0709] Example 2: Using the Conference Playbook

[0710] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook that includes the following questions: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also uses an emotion engine to analyze the emotional tendencies of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure the meeting proceeds smoothly.

[0711] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0712] The processing flow will be explained below.

[0713] Step 1:

[0714] A user speaks during a conference, and audio data is collected by the device's microphone.

[0715] Step 2:

[0716] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[0717] Step 3:

[0718] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[0719] Step 4:

[0720] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to identify the topic and importance of the comment.

[0721] Step 5:

[0722] The server passes the analysis results obtained by the NLP engine to the emotion engine, which recognizes the speaker's emotions. The emotion engine evaluates the type of emotion (e.g., joy, anger, anxiety, etc.) and intensity of the emotion contained in the speech.

[0723] Step 6:

[0724] The server monitors the progress of the meeting based on the analysis results and emotion data, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[0725] Step 7:

[0726] The server generates appropriate advice and suggestions based on the abnormal progress and emotion data, such as "You might want to ask for more opinions on this topic" or "Participant C is showing anxiety, so ask for their opinion."

[0727] Step 8:

[0728] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[0729] Step 9:

[0730] The device displays the received advice, suggestions, and emotion-related feedback to the user through a GUI (Graphical User Interface).

[0731] Step 10:

[0732] Users can view advice, suggestions, and emotional feedback displayed on their device screen and use it to guide the meeting, for example by asking specific participants for their opinions or changing the focus of the discussion.

[0733] Step 11:

[0734] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[0735] Step 12:

[0736] The server sends the generated meeting playbook and the results of sentiment analysis to the device, making them available before and during the meeting.

[0737] Step 13:

[0738] The device displays the received meeting playbook and sentiment analysis results, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[0739] Specific examples

[0740] Example 1: Real-time feedback

[0741] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine. The emotion engine then recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for participant B's opinion," and send it to the device in real time. The device displays this, and the user can respond to the suggestion by saying, "Before that, Mr. B, do you have any comments?", thereby ensuring the smooth progress of the meeting.

[0742] Example 2: Using the Conference Playbook

[0743] The server analyzes past meeting data and learns patterns using generative artificial intelligence. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It then uses an emotion engine to analyze the emotional trends of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. The user can use this as a reference to ask effective questions and keep the meeting running smoothly.

[0744] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0745] Example 2

[0746] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0747] Conventional meeting support systems did not take into account the emotions of participants, which resulted in problems such as meetings stalling or making it difficult for some participants to speak up. Furthermore, they lacked the functionality to monitor meeting progress in real time and provide appropriate advice and suggestions, making it difficult to facilitate effective discussions. Furthermore, there were insufficient means to effectively utilize past meeting data, which often led to repeated discussions of the same issues.

[0748] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0749] In this invention, the server includes: means for receiving voice data and converting it into text data using an automatic speech recognition engine; means for analyzing it using a natural language processing engine; means for identifying the user's emotions based on the analysis results, monitoring the progress of the meeting, and detecting abnormal progress; means for generating advice and suggestions based on the detection results and the identified emotions and transmitting the suggestions to the terminal; and means for analyzing past meeting data, generating a meeting playbook using a generative AI model, and transmitting the playbook to the terminal. This enables real-time support for the progress of meetings and the provision of advice and suggestions that take participants' emotions into consideration. Furthermore, past meeting data can be utilized to generate and display a meeting playbook to guide effective discussions.

[0750] "Audio data" refers to recorded information of words or remarks made by users during a conference.

[0751] "Server" refers to a computer system that receives, analyzes, identifies emotions, monitors, generates advice, and generates meeting playbooks for voice data.

[0752] "Terminal" refers to a computer or smart device used by a user that collects voice data and displays advice, suggestions, and meeting playbooks sent from the server.

[0753] An "automatic speech recognition engine" refers to software technology for analyzing voice data and converting it into text data.

[0754] "Text data" refers to character information converted from voice data by an automatic voice recognition engine.

[0755] A "natural language processing engine" refers to a technology that analyzes text data, understands its content and context, and extracts information.

[0756] "Emotion engine" refers to software technology that identifies a user's emotions based on text data.

[0757] "Monitoring" refers to the process of observing the progress of a meeting in real time and detecting any anomalies or problems.

[0758] "Advice" refers to recommendations and suggestions generated by the server to improve the progress of the meeting and the quality of the discussion.

[0759] "Suggestion" refers to a specific solution or instruction for action provided by the server for a particular situation or problem.

[0760] A "generative AI model" refers to an artificial intelligence model that learns from past data and makes predictions and generates based on new data.

[0761] "Meeting playbooks" refer to templates and guidelines for conducting effective discussions, created by a generative AI model based on past meeting data.

[0762] "Real-time display" refers to the immediate display of received advice or suggestions on the device.

[0763] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, this system makes meetings more effective by combining it with an emotion engine that recognizes the user's emotions. The specific form of this system is shown below.

[0764] Overall system configuration

[0765] This system is roughly divided into three components: a server, a terminal, and a user. The server has an emotion engine built in.

[0766] Server Features

[0767] The server collects, analyzes, monitors, and generates suggestions for voice data. Specifically, the server receives voice data and converts it into text using an automatic speech recognition engine such as Google Cloud Speech-to-Text. It then analyzes the text data using a natural language processing engine such as GPT-4 to identify the topic and sentiment of the speech. It then uses an emotion engine such as IBM Watson Tone Analyzer to recognize the user's sentiment from the analysis results. This allows the server to monitor the progress of the meeting and detect any abnormalities.

[0768] If an anomaly is detected, the server generates advice or suggestions using a generative AI model (e.g., GPT-3) and sends them to the device in real time. The server also analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). This playbook provides templates to help facilitate effective discussions.

[0769] Device Features

[0770] The terminal is a computer or smart device used by the user that collects voice data and sends it to the server. The terminal also displays real-time advice and suggestions sent from the server and provides them to the user. It also receives and displays the generated meeting playbook and sentiment-related feedback. This can be achieved through a custom GUI in a web browser or a plugin for the meeting software (e.g., Zoom, Microsoft Teams).

[0771] User Roles

[0772] Users are participants in a meeting and generate voice data by speaking. This voice data is sent to the server via their devices. Users refer to the advice and suggestions sent from the server to guide the meeting appropriately. Users can also refer to the generated meeting playbook to receive support for effective question and answer sessions and discussions.

[0773] Specific examples

[0774] Example 1: Real-time feedback

[0775] User A says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using Google Cloud Speech-to-Text. Next, it analyzes the text data using the natural language processing engine GPT-4 and recognizes the speaker's emotions using IBM Watson Tone Analyzer. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server generates a suggestion such as "Before that, please ask for participant B's opinion," and sends feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who then uses them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[0776] Prompt Sentence Examples

[0777] "User A wants to move on to the next agenda item, but seems impatient. Suggest that they listen to Participant B's opinion."

[0778] Example 2: Using the Conference Playbook

[0779] The server analyzes past project management meeting data and learns patterns using a generative AI model (e.g., BERT). For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also analyzes the emotional trends of participants in past meetings using IBM Watson Tone Analyzer. The server then sends the generated playbook and advice based on the emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure smooth progress in meetings.

[0780] Prompt Sentence Examples

[0781] "This is a meeting playbook generated from past meeting data. Please ask questions and make suggestions to elicit opinions from your team members."

[0782] This system combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0783] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0784] Step 1: Collecting audio data

[0785] The device collects the voice data of the user during the conference. The voice data is recorded through the device's microphone. The input is the user's speech, and the output is the recorded voice data. Specifically, the conference application captures the voice data in real time and prepares it to be sent to the server.

[0786] Step 2: Sending the audio data to the server

[0787] The device sends the collected voice data to the server. The input is the recorded voice data, and the output is the voice data sent to the server. Specifically, the device's communication module is used to encrypt the voice data and send it to the server via the network.

[0788] Step 3: Converting audio data to text

[0789] The server converts the received voice data into text data using an automatic speech recognition engine such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data. Specifically, the automatic speech recognition engine extracts words and sentences from the voice and outputs them as structured text data.

[0790] Step 4: Analyzing the text data

[0791] The server analyzes the text data using a natural language processing engine such as GPT-4. The input is the text data, and the output is the analyzed text data and its topic and context information. Specifically, the natural language processing engine interprets the content of the text data, extracts important keywords and phrases, and classifies them according to topic and context.

[0792] Step 5: Identify the emotion

[0793] The server uses an emotion engine such as IBM Watson Tone Analyzer to identify the user's emotion from the analyzed text data. The input is the analyzed text data, and the output is the identified emotion information. Specifically, the emotion engine analyzes the wording and structure in the text to identify the user's emotional state (e.g., joy, sadness, anger).

[0794] Step 6: Monitor the progress of the meeting

[0795] The server monitors the progress of the meeting in real time based on the identified emotions and text data. The input is emotion information and analyzed text data, and the output is the current progress of the meeting and the detection results of abnormal progress. Specifically, the server tracks this data and analyzes the frequency and content of comments, participants' reactions, etc.

[0796] Step 7: Generate advice and suggestions

[0797] The server generates advice or suggestions using a generative AI model (e.g., GPT-3) based on the detection results and identified emotions. The input is the abnormal progress detection results and emotion information, and the output is text data of the advice or suggestions. Specifically, the server forms specific suggestions such as "In this situation, please say this next" based on the generative AI model.

[0798] Step 8: Submit and view tips and suggestions

[0799] The server sends the generated advice or suggestion to the terminal, which displays it in real time. The input is text data of the advice or suggestion, and the output is the advice or suggestion displayed on the terminal. Specifically, it is displayed to the user as a pop-up window or a notification message using the terminal's display device.

[0800] Step 9: Analyze historical data and generate meeting playbooks

[0801] The server analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). The input is past meeting data, and the output is the generated meeting playbook. Specifically, the generative AI model learns from the data, extracts patterns of successful discussions and effective questions, and compiles them into templates.

[0802] Step 10: Submit and view the meeting playbook

[0803] The server sends the generated conference playbook to the terminal, which receives and displays it. The input is the generated conference playbook, and the output is the conference playbook displayed on the terminal. Specifically, it is provided to the user as a template or guideline displayed on the GUI.

[0804] (Application example 2)

[0805] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0806] Current food delivery services lack a system that can quickly respond to the difficulties and emotional changes that delivery workers face in the field. This can easily lead to delivery delays and reduced customer satisfaction. To provide an environment where delivery workers can perform their work efficiently and with peace of mind, a system is needed that monitors the progress of deliveries in real time and provides appropriate support and suggestions.

[0807] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0808] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data received by the server into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of delivery based on the results of the analysis by the server and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the advice or suggestions received in real time, means for the server to analyze past delivery data and generate a delivery playbook using generative artificial intelligence and transmit it to the terminal, and means for the terminal to display the delivery playbook received. This enables delivery personnel to respond quickly and appropriately at the scene.

[0809] "Voice data" is information collected in digital form from the voices of delivery personnel.

[0810] "Real-time" means that data is processed immediately, without delay.

[0811] The "server" is a central system that receives and analyzes voice data and sends appropriate suggestions and feedback to the device.

[0812] An "automatic speech recognition engine" is a technology for converting voice data into text data.

[0813] "Text data" is character information converted by a voice recognition engine.

[0814] A "natural language processing engine" is a technology for analyzing text data and identifying meaning and emotion.

[0815] "Progress" refers to the current status in the delivery process.

[0816] "Monitoring" means watching progress in real time.

[0817] "Abnormal" refers to a situation where delivery is not proceeding as planned.

[0818] "Advice" is a suggestion of specific action to the delivery person.

[0819] A "suggestion" is advice for the delivery person to improve the current situation.

[0820] A "terminal" is a receiving device carried by a delivery person.

[0821] "Historical delivery data" is a collection of information about deliveries that have been made previously.

[0822] "Generative artificial intelligence" is a technology for generating new insights and suggestions from data.

[0823] The "Delivery Playbook" is a guidebook created based on past delivery data to support efficient delivery.

[0824] "Display" means providing information visually on a terminal.

[0825] This invention relates to a system that supports food delivery operations. In particular, it is a system for realizing a smart device application that recognizes the emotions and situations of delivery personnel in real time and provides appropriate advice and suggestions. This system is broadly composed of three elements: a server, a terminal, and a delivery personnel. The specific form of the system is shown below.

[0826] Overall system configuration

[0827] This system sends the delivery person's voice data to a server in real time, where it is converted into text data using an automatic speech recognition engine. The voice data is then analyzed using a natural language processing engine to monitor the progress of the delivery. If an abnormality is detected, the server generates appropriate advice or suggestions and sends them to the terminal. The terminal displays these advice and suggestions in real time, helping the delivery person quickly choose their next action. The server also analyzes past delivery data and uses generative artificial intelligence to generate a delivery playbook, which is sent to the terminal. The delivery playbook serves as a guideline for efficiently carrying out delivery work.

[0828] Hardware and software used

[0829] Hardware

[0830] Smartphone: A device carried by delivery personnel that collects voice data and displays advice and suggestions.

[0831] Server: A central system that analyzes data, generates proposals, and analyzes historical data.

[0832] software

[0833] Automatic speech recognition engine: Converts voice data into text data (e.g., Google Cloud Speech-to-Text).

[0834] Natural language processing engines: Analyze text data and identify meaning and sentiment (e.g., spaCy, NLTK).

[0835] Emotion Recognition AI: Recognize emotions from text data (e.g., Microsoft Azure Emotion API).

[0836] Generative AI: Generate new insights and suggestions from data (e.g., OpenAI GPT-4).

[0837] Program processing

[0838] Server processing

[0839] The server receives voice data sent by the delivery person. It then uses an automatic speech recognition engine to convert the voice data into text data. The converted text data is analyzed by a natural language processing engine to identify the content and emotion of the speech. Emotion recognition AI is used to recognize the emotions of the delivery person and monitor the delivery progress. If abnormal progress is detected, the server uses generation AI to generate appropriate advice or suggestions and send them to the device.

[0840] Specific example explanation

[0841] Example 1: Real-time support

[0842] When a delivery person says, "I'm going to be late," this voice data is sent from the smartphone to a server. The server receives the voice data and converts it into text using Google Cloud Speech-to-Text. The text data is analyzed using spaCy, and the keyword "I'm going to be late" is extracted. When analyzed using the Microsoft Azure Emotion API, it is identified that the delivery person is feeling anxious. The suggestion generated using OpenAI GPT-4 is the message, "Please check if there is a detour nearby. Also, if congestion persists, we recommend that you report the delay to customers and take a break." This message is immediately displayed on the delivery person's smartphone. Below is an example of a prompt.

[0843] Example prompt: What is the next action the delivery person should take if "It looks like it's going to be late" and "The roads are busy"? Generate a feedback statement.

[0844] In this way, delivery personnel can respond quickly and appropriately on site.

[0845] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0846] Step 1:

[0847] During delivery, the user inputs voice data into the smartphone, for example, "I'm going to be late, the roads are congested." This voice data is collected and sent from the smartphone to the server in real time.

[0848] Input: Delivery person's voice data

[0849] Output: Audio data sent to the server

[0850] Step 2:

[0851] The server converts the received voice data into text data using an automatic speech recognition engine (Google Cloud Speech-to-Text). Through this process, the voice data is converted into text information.

[0852] Input: Audio data

[0853] Output: Text data

[0854] Step 3:

[0855] The server analyzes the obtained text data using a natural language processing engine (spaCy or NLTK). Keywords in the text data (e.g., "I'm going to be late" or "The roads are congested") are extracted, and the topic and content of the utterance are identified.

[0856] Input: Text data

[0857] Output: Parsed topic and content information

[0858] Step 4:

[0859] The server uses emotion recognition AI (Microsoft Azure Emotion API) to recognize the emotions of the delivery person from the analysis results. Emotion recognition determines whether the delivery person is feeling anxious or stressed from the text data.

[0860] Input: Parsed topic and content information

[0861] Output: Recognized emotion data

[0862] Step 5:

[0863] The server monitors the delivery progress based on the analysis results and sentiment data, and if an abnormal progress (e.g., risk of delay) is detected, it proceeds to the next step.

[0864] Input: Recognized emotion data

[0865] Output: Progress determination result

[0866] Step 6:

[0867] The server uses a generative AI model (OpenAI GPT-4) to generate appropriate advice and suggestions. The prompt sentence is set as "What is the next action the delivery person should take if the delivery person is likely to be late and the roads are congested?" and the generated suggestions are created.

[0868] Input: Progress status determination result

[0869] Output: Generated advice and suggestions

[0870] Step 7:

[0871] The server sends the generated advice and suggestions to the terminal.

[0872] Input: Generated advice and suggestions

[0873] Output: Tips and suggestions sent to the terminal

[0874] Step 8:

[0875] The device displays the received advice and suggestions in real time. The delivery person can then refer to these advice and suggestions and select the appropriate action to carry out their task. Specifically, they can check for detours according to the example sentences displayed, or report delays to the customer.

[0876] Input: Tips and suggestions sent to your device

[0877] Output: Tips and suggestions displayed on the terminal

[0878] This allows delivery personnel to respond quickly and appropriately on site.

[0879] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0880] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0881] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0882] [Third embodiment]

[0883] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0884] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0885] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0886] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0887] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0888] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0889] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0890] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0891] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0892] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0893] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0894] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0895] The present invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. The specific form of this system is shown below.

[0896] Overall system configuration

[0897] This system is broadly composed of three elements: a server, a terminal, and a user.

[0898] server

[0899] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and sentiment of the speech. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates advice and suggestions based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0900] Terminal

[0901] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. It also receives and displays the generated meeting playbook for the user to refer to.

[0902] User

[0903] Users are participants in a conference and make comments during the conference. The voice data of the user's speech is sent to the server via the terminal, where it is analyzed. The user can refer to the advice and suggestions displayed on the terminal to make comments at the appropriate time and to proceed with the conference.

[0904] Program processing

[0905] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and sentiment of the comments. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[0906] Specific examples

[0907] Example 1: Real-time feedback

[0908] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the progress is monitored. If a topic that requires discussion has been skipped, the server generates a suggestion such as "Before moving on, please ask participant B for his opinion," and sends it to the device. The device receives this suggestion and displays it to the user, who then proceeds with the meeting by saying, "Before that, Mr. B, do you have any comments?"

[0909] Example 2: Using the Conference Playbook

[0910] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" The server sends the generated playbook to the device, which receives it and displays it on a GUI. Users can refer to the playbook to ask effective questions and keep the meeting running smoothly.

[0911] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[0912] The processing flow will be explained below.

[0913] Step 1:

[0914] A user speaks during a conference, and audio data is collected by the device's microphone.

[0915] Step 2:

[0916] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[0917] Step 3:

[0918] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[0919] Step 4:

[0920] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to determine the topic, sentiment, and importance of the utterance.

[0921] Step 5:

[0922] The server monitors the progress of the meeting based on the analysis results, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[0923] Step 6:

[0924] If the server detects abnormal progress, it generates appropriate advice or suggestions, such as "You might want to seek further feedback on this topic."

[0925] Step 7:

[0926] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[0927] Step 8:

[0928] The advice and suggestions received by the device are displayed in a format that provides a good user experience (UX), presented through a GUI (Graphical User Interface).

[0929] Step 9:

[0930] Users can check the advice and suggestions displayed on their device screen and proceed with the meeting based on them, for example, by asking specific participants for their opinions or changing the focus of the discussion.

[0931] Step 10:

[0932] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[0933] Step 11:

[0934] The server generates a meeting playbook and sends it to the device, making it available before and during the meeting.

[0935] Step 12:

[0936] The terminal displays the received meeting playbook, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[0937] The process flow of this system will help meetings run smoothly and improve the quality of discussions, allowing all participants to participate effectively, greatly increasing meeting efficiency.

[0938] Example 1

[0939] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0940] Improving the progress of meetings and the quality of discussions is a crucial issue in modern organizations. However, meetings often do not proceed as planned, discussions go off track, and some participants find it difficult to speak up. Furthermore, there is a lack of methods to effectively utilize past meeting data to conduct meetings effectively. To solve these issues, real-time progress management and appropriate feedback to participants are required.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0942] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the received voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the conference based on the analysis results and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the received advice or suggestions in real time, means for the server to analyze past conference data and generate a conference playbook using artificial intelligence and transmit it to the terminal, and means for the terminal to display the received conference playbook. This makes it possible to manage the progress of the conference in real time and provide appropriate advice immediately when an abnormal situation is detected. Furthermore, past conference data can be used to generate effective playbooks, improving the quality and efficiency of conferences.

[0943] "Voice data" refers to data for recording and transmitting user speech and voice information in digital format.

[0944] "Server" means a computer system that receives, analyzes, monitors, generates, and transmits audio data.

[0945] A "terminal" is a computer or smart device used by a user, and is a device for collecting voice data and displaying advice and suggestions from the server.

[0946] A "user" is a participant in a conference, who speaks and receives advice and suggestions from the server.

[0947] An "automatic speech recognition engine" is a software component for converting voice data into text data.

[0948] A "natural language processing engine" is a software component that analyzes text data and identifies meaning and sentiment.

[0949] "Monitoring" is the act or process of watching the progress of a meeting in real time and detecting anomalies.

[0950] An "abnormal progress" is when a meeting deviates from the planned course of action, certain topics are skipped, or necessary discussions are not held.

[0951] "Advice" refers to instructions or suggestions provided to facilitate the smooth running of a meeting.

[0952] A "proposal" is a specific action or instruction to solve a problem.

[0953] A "meeting playbook" is a document generated based on past meeting data that includes materials and sample questions for conducting effective meetings.

[0954] "Real-time" refers to processing or communication that occurs nearly immediately or without delay.

[0955] "Generative AI" is an artificial intelligence technology that analyzes large amounts of data, learns patterns and knowledge, and generates new information and suggestions.

[0956] System configuration

[0957] The conference support system of the present invention is composed of three main elements: a server, a terminal, and a user.

[0958] server

[0959] The server is responsible for receiving, analyzing, and monitoring the audio data, and generating advice and suggestions. Specifically, the process proceeds as follows:

[0960] Receiving and converting audio data

[0961] The server receives voice data from the device, converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API), and sends the text data to a natural language processing engine (e.g., OpenAI's GPT-3) for analysis.

[0962] Text data analysis

[0963] The server then analyzes the converted text data using a natural language processing engine to identify the topic and sentiment of the comments, and monitors the progress of the meeting in real time based on the analysis results to detect any anomalies.

[0964] Generating advice and suggestions

[0965] If the server detects an abnormality in the progress, it generates appropriate advice or suggestions. For example, it may generate a specific suggestion such as, "Please ask participant B for his / her opinion before moving on to the next agenda item." This is generated using a generative AI model (e.g., GPT-3).

[0966] Generate a conference playbook

[0967] The server analyzes past meeting data and generates a meeting playbook to support effective progress. The playbook is a document that learns and summarizes past patterns and effective questions.

[0968] Terminal

[0969] The device collects user utterances, transmits the audio data to the server, and displays advice, suggestions, and meeting playbooks received from the server in real time.

[0970] Collection and transmission of voice data

[0971] The device collects the user's speech using a microphone and transmits it to the server in real time using WebSocket or HTTP POST requests.

[0972] View advice, suggestions and playbooks

[0973] Advice and suggestions received from the server are displayed in real time on the terminal, possibly as a pop-up window or in a notification area. The meeting playbook sent from the server is also displayed on the GUI for the user to refer to in real time.

[0974] User

[0975] Users are participants in the conference and make comments. Their comments are collected by their devices and sent to the server. The progress of the conference is adjusted based on advice and suggestions displayed on the devices.

[0976] User comments and reviews

[0977] The user makes a statement and reviews the advice and suggestions displayed on the device, which determines what to say next and how to proceed.

[0978] Specific examples

[0979] Real-time feedback

[0980] 1. A user says, "Let's move on to the next item on the agenda."

[0981] 2. The device collects the voice data and sends it to the server.

[0982] 3. The server converts the voice data into text data using the Google Speech-to-Text API.

[0983] 4. The server analyzes the text data using OpenAI's GPT-3 and monitors progress.

[0984] 5. The server determines that a topic requiring discussion has been skipped and generates an advice.

[0985] 6. The server sends the advice to the terminal: "Please ask participant B for his opinion before moving on."

[0986] 7. The device receives the advice and displays it to the user in a pop-up notification.

[0987] 8. The user says, "Before that, Mr. B, do you have any comments?"

[0988] Using the Conference Playbook

[0989] 1. The server analyzes past meeting data and generates a meeting playbook containing effective questions.

[0990] 2. The server sends the generated conference playbook to the terminal.

[0991] 3. The terminal receives the conference playbook and displays it on the GUI.

[0992] 4. The user consults the meeting playbook and schedules the next question to be asked.

[0993] 5. Users effectively facilitate meetings by asking the question, "What's the progress on risk management?"

[0994] Prompt Sentence Examples

[0995] "Generate a reminder to avoid skipping topics that need discussion after saying let's move on to the next agenda item."

[0996] "Generate a playbook with useful questions for project management meetings."

[0997] In this way, the system of the present invention can support the progress of meetings and improve the quality of discussions by providing real-time feedback and utilizing past data.

[0998] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0999] System program processing flow

[1000] Server Processing Steps

[1001] Step 1: Receiving audio data

[1002] Step 2: Convert the audio data

[1003] Step 3: Analyzing the text data

[1004] Step 4: Monitor the progress of the meeting

[1005] Step 5: Generate advice and suggestions

[1006] Step 6: Submit your advice and suggestions

[1007] Step 7: Generate the conference playbook

[1008] Terminal processing steps

[1009] Step 1: Collecting audio data

[1010] Step 2: Sending audio data

[1011] Step 3: Receiving advice and suggestions

[1012] Step 4: Viewing Advice and Suggestions

[1013] Step 5: Receive the conference playbook

[1014] Step 6: View the conference playbook

[1015] User processing steps

[1016] Step 1: Speak up

[1017] Step 2: Review advice and suggestions

[1018] Step 3: Refer to the conference playbook

[1019] Detailed explanation of the processing steps

[1020] Server Processing Steps

[1021] Step 1: Receiving audio data

[1022] The server receives audio data from the device. The input is the audio data sent from the device, and the output is the received audio data. This audio data is received via WebSocket or HTTP POST request.

[1023] Step 2: Convert the audio data

[1024] The server converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API). The input is the received voice data, and the output is the converted text data. Converting the voice data into text data prepares it for natural language processing.

[1025] Step 3: Analyzing the text data

[1026] The server analyzes the converted text data using a natural language processing engine (e.g., OpenAI's GPT-3). The input is text data, and the output is the analysis results. The analysis results include the topic and sentiment of the comments. This analysis allows for a concrete understanding of the progress of the meeting.

[1027] Step 4: Monitor the progress of the meeting

[1028] The server monitors the progress of the conference based on the results of analyzing the text data. The input is the analysis results, and the output is monitoring information. This information is used to detect abnormal progress (such as skipping topics or delays in progress).

[1029] Step 5: Generate advice and suggestions

[1030] If the server detects abnormal progress, it generates advice or suggestions. The input is monitoring information, and the output is the generated advice or suggestions. A generative AI model (e.g., GPT-3) is used to suggest specific actions. For example, it generates suggestions such as, "Please ask participant B for his opinion before moving on to the next agenda item."

[1031] Step 6: Submit your advice and suggestions

[1032] The server sends generated advice and suggestions to the device. The input is the generated advice and suggestions, and the output is the sent advice and suggestions. The low-latency WebSocket protocol is used to notify the device in real time.

[1033] Step 7: Generate the conference playbook

[1034] The server analyzes past meeting data and generates a meeting playbook. The input is past meeting data, and the output is the generated meeting playbook. A generative AI model is used to learn past patterns and effective example questions, and generate a specific playbook.

[1035] Terminal processing steps

[1036] Step 1: Collecting audio data

[1037] The device collects the user's speech from a microphone. The input is the user's speech, and the output is the collected voice data. This voice data is sent to the server in real time.

[1038] Step 2: Sending audio data

[1039] The device sends collected audio data to the server. The input is the collected audio data, and the output is the audio data sent to the server. Data is sent with low latency using WebSocket or HTTP POST requests.

[1040] Step 3: Receiving advice and suggestions

[1041] The device receives advice and suggestions sent from the server. The input is the advice and suggestions sent from the server, and the output is the received advice and suggestions. The advice and suggestions are received in real time and reflected in the next action.

[1042] Step 4: Viewing Advice and Suggestions

[1043] The advice and suggestions received by the device are displayed in real time on the GUI. The input is the received advice and suggestions, and the output is the displayed advice and suggestions. The user is notified using a pop-up window or notification area.

[1044] Step 5: Receive the conference playbook

[1045] The terminal receives the meeting playbook sent from the server. The input is the meeting playbook sent from the server, and the output is the received meeting playbook. It is received in real time and used during the meeting.

[1046] Step 6: View the conference playbook

[1047] The terminal displays the received conference playbook on the GUI. The input is the received conference playbook, and the output is the displayed conference playbook, allowing the user to refer to it immediately.

[1048] User processing steps

[1049] Step 1: Speak up

[1050] Users make comments during a conference. The input is the progress of the conference, and the output is the user's comments. These comments are collected by the terminal and sent to the server.

[1051] Step 2: Review advice and suggestions

[1052] The user reviews the advice and suggestions displayed on the device. The input is the displayed advice and suggestions, and the output is the user's review. The next action is determined based on the review result.

[1053] Step 3: Refer to the conference playbook

[1054] The user refers to the meeting playbook received from the server. The input is the displayed meeting playbook, and the output is the next action based on the reference. The playbook is used to adjust the content and timing of remarks.

[1055] In this way, the elements of the server, terminals, and users work together to realize a system that improves the progress of meetings and the quality of discussions.

[1056] (Application example 1)

[1057] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1058] Conventional meeting support systems lack effective means for improving meeting progress and the quality of discussions. In particular, in production meetings in factories and other situations where real-time feedback is required, efficient and prompt progress is essential, but current systems are insufficient. Furthermore, because they are unable to perform sentiment analysis during ongoing meetings, it is difficult to accurately detect problems that arise during meetings and provide advice and suggestions for improvement. Therefore, a new system is needed to improve the efficiency and quality of meetings.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1060] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the meeting based on the results and detecting abnormal progress, means for generating advice and suggestions and transmitting them to the terminal, means for identifying emotions and generating appropriate advice and suggestions based thereon and transmitting them to the terminal, means for generating a meeting playbook and transmitting it to the terminal, and means for displaying the advice and suggestions based on the emotions received by the terminal in real time. This makes it possible to provide appropriate feedback based on emotions in production meetings and other meetings and effectively support the progress of the meeting.

[1061] "Audio data" refers to data in which the sound of a meeting, conversation, etc. is recorded in digital format.

[1062] "Server" is a computer system for collecting, analyzing, monitoring, and generating suggestions from speech data.

[1063] An "automatic speech recognition engine" is a technology that converts voice data into text data.

[1064] A "natural language processing engine" is a technology for analyzing text data and understanding its content.

[1065] "Emotion identification" is a technology that determines the emotion of a speaker based on text data.

[1066] "Real-time" refers to processing occurring immediately without delay.

[1067] "Progress monitoring" is a technique for constantly monitoring and recording the progress of meetings and work.

[1068] "Anomaly detection" refers to discovering when progress is not normal.

[1069] "Advice and suggestion generation" is a technology that generates advice and suggestions to support the progress of meetings and work based on the analysis results.

[1070] A "meeting playbook" is a guide document generated by analyzing past meeting data to support the progress of the next meeting.

[1071] A "terminal" is a device that receives advice, suggestions, and conference playbooks sent from the server and displays them to the user.

[1072] The system of the present invention is characterized by collecting voice data in real time and analyzing it on a server in order to improve the progress of meetings and the quality of discussions. Furthermore, advice and suggestions are generated based on the analysis results, and are sent to and displayed on terminals, thereby supporting efficient progress of meetings.

[1073] Overall system configuration

[1074] This system is broadly composed of three elements: a server, a terminal, and a user.

[1075] server

[1076] The server collects, analyzes, monitors, and generates suggestions from audio data. It uses the following hardware and software:

[1077] Hardware: A server computer equipped with a high-performance processor

[1078] Software: Google Cloud Speech-to-Text API, automatic speech recognition engine, TextBlob, natural language processing engine

[1079] The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed using a natural language processing engine to identify the topic and sentiment of the speech. This allows the progress of the meeting to be monitored and abnormal progress to be detected. For example, when a speech such as "Let's move on to the next agenda item" is heard using text analysis, the system checks based on this content to see if necessary topics have been skipped. If necessary, it generates a suggestion such as "Please ask participants for their opinions before moving on."

[1080] Terminal

[1081] A terminal is a computer or smart device used by a user that has the following functions:

[1082] Audio data collection: Use a microphone to collect meeting audio

[1083] Data transmission: Collected voice data is sent to the server in real time.

[1084] Data display: Real-time display of advice, suggestions, and meeting playbooks sent from the server

[1085] The terminal receives suggestions from the server and displays them to the user, for example, "Please review your risk management before proceeding to the next step" based on the meeting playbook.

[1086] User

[1087] Users are participants in a conference and make comments during the conference. The voice data spoken by the user is sent to the server via the terminal. The results of analysis by the server are displayed on the terminal, and the user can use this information to proceed with the conference. For example, by saying, "Before we begin, do you have any comments?", the ongoing discussion can be stimulated.

[1088] Specific examples

[1089] When a user says, "There is a delay in the next production line," the voice data is sent to the server via the device. The server converts the voice data into text data using an automatic speech recognition engine and analyzes it using TextBlob. Based on the analysis results, the server generates a suggestion saying, "There appears to be a delay. Let's identify the specific problem," and sends it to the device. The device receives this suggestion and displays it to the user.

[1090] Prompt Sentence Examples

[1091] Input: "The next production line is experiencing delays."

[1092] output:

[1093] Voice recognition result: There is a delay in the next production line

[1094] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[1095] Tip: It looks like you're having a problem. Let's find out what the problem is.

[1096] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1097] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1098] Step 1:

[1099] This system collects voice data spoken by users during a conference. The input is the voice spoken by users during a conference. The device collects this voice using a built-in microphone or an external microphone and records it as digital voice data. The output is the collected digital voice data.

[1100] Step 2:

[1101] The collected voice data is sent to the server in real time. The input is the digital voice data collected in step 1. The device sends this voice data to the server via the network. The output is the voice data received by the server.

[1102] Step 3:

[1103] The server converts the received voice data into text data using an automatic speech recognition engine. The input is the voice data received by the server. The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The output is the converted text data.

[1104] Step 4:

[1105] The server analyzes the text data using a natural language processing engine to identify the topic and sentiment of the utterance. The input is the text data generated in step 3. The server analyzes the text data using TextBlob to identify the topic and sentiment. Specifically, it extracts important keywords from the text and performs sentiment analysis. The output is the topic and sentiment data as the analysis results.

[1106] Step 5:

[1107] The server monitors the progress of the meeting based on the analysis results and detects abnormal progress. The input is the topic and emotion data obtained in step 4. The server monitors the progress in real time and detects abnormal situations, such as when an important topic in the meeting is skipped. The output is the detection result when an abnormality is detected.

[1108] Step 6:

[1109] The server generates advice and suggestions based on the detection results and sends them to the terminal. The input is the anomaly detection result from step 5. The server generates advice and suggestions for the next step based on the analysis results and detection results and sends these advice and suggestions to the terminal. The output is the generated advice and suggestions.

[1110] Step 7:

[1111] The device displays the received advice or suggestion to the user in real time. The input is the advice or suggestion generated in step 6. The device displays this advice or suggestion on the screen and notifies the user. The output is the advice or suggestion that the user can see on the screen.

[1112] Step 8:

[1113] The server analyzes past meeting data and uses generative AI to generate a meeting playbook, which is then sent to the device. The input is the previously accumulated meeting data. The server uses a generative AI model to learn past patterns and effective questions, and generates the meeting playbook. The output is the generated meeting playbook.

[1114] Step 9:

[1115] The terminal displays the received conference playbook. The input is the conference playbook generated in step 8. The terminal displays the generated playbook on the screen so that the user can refer to it. The output is a conference playbook that the user can check.

[1116] Examples of prompt statements

[1117] Input: "The next production line is experiencing delays."

[1118] output:

[1119] Voice recognition result: There is a delay in the next production line

[1120] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[1121] Tip: It looks like you're having a problem. Let's find out what the problem is.

[1122] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1123] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, by combining it with an emotion engine that recognizes the user's emotions, the progress of meetings can be made more effective. Specific examples of this system are shown below.

[1124] Overall system configuration

[1125] This system is broadly composed of three elements: the server, the terminal, and the user. The server also has an emotion engine built in.

[1126] server

[1127] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. It then uses an emotion engine to recognize the user's emotions from the analysis results. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates appropriate advice and suggestions that take emotions into account based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[1128] Terminal

[1129] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. The terminal also receives and displays the generated meeting playbook and emotion-related feedback for the user to refer to.

[1130] User

[1131] Users are participants in a conference and speak during the conference. The voice data they speak is sent to the server via their device, where it is analyzed. They use the advice, suggestions, and emotion-related feedback displayed on their device to speak at the appropriate time and guide the conference.

[1132] Program processing

[1133] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. Finally, it uses an emotion engine to recognize the user's emotion from the analysis results. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time, and emotional feedback is also provided. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[1134] Specific examples

[1135] Example 1: Real-time feedback

[1136] The user says, "Let's move on to the next item on the agenda." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the emotion engine recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for Participant B's opinion," and send feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who can then use them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[1137] Example 2: Using the Conference Playbook

[1138] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook that includes the following questions: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also uses an emotion engine to analyze the emotional tendencies of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure the meeting proceeds smoothly.

[1139] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1140] The processing flow will be explained below.

[1141] Step 1:

[1142] A user speaks during a conference, and audio data is collected by the device's microphone.

[1143] Step 2:

[1144] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[1145] Step 3:

[1146] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[1147] Step 4:

[1148] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to identify the topic and importance of the comment.

[1149] Step 5:

[1150] The server passes the analysis results obtained by the NLP engine to the emotion engine, which recognizes the speaker's emotions. The emotion engine evaluates the type of emotion (e.g., joy, anger, anxiety, etc.) and intensity of the emotion contained in the speech.

[1151] Step 6:

[1152] The server monitors the progress of the meeting based on the analysis results and emotion data, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[1153] Step 7:

[1154] The server generates appropriate advice and suggestions based on the abnormal progress and emotion data, such as "You might want to ask for more opinions on this topic" or "Participant C is showing anxiety, so ask for their opinion."

[1155] Step 8:

[1156] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[1157] Step 9:

[1158] The device displays the received advice, suggestions, and emotion-related feedback to the user through a GUI (Graphical User Interface).

[1159] Step 10:

[1160] Users can view advice, suggestions, and emotional feedback displayed on their device screen and use it to guide the meeting, for example by asking specific participants for their opinions or changing the focus of the discussion.

[1161] Step 11:

[1162] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[1163] Step 12:

[1164] The server sends the generated meeting playbook and the results of sentiment analysis to the device, making them available before and during the meeting.

[1165] Step 13:

[1166] The device displays the received meeting playbook and sentiment analysis results, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[1167] Specific examples

[1168] Example 1: Real-time feedback

[1169] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine. The emotion engine then recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for participant B's opinion," and send it to the device in real time. The device displays this, and the user can respond to the suggestion by saying, "Before that, Mr. B, do you have any comments?", thereby ensuring the smooth progress of the meeting.

[1170] Example 2: Using the Conference Playbook

[1171] The server analyzes past meeting data and learns patterns using generative artificial intelligence. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It then uses an emotion engine to analyze the emotional trends of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. The user can use this as a reference to ask effective questions and keep the meeting running smoothly.

[1172] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1173] Example 2

[1174] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1175] Conventional meeting support systems did not take into account the emotions of participants, which resulted in problems such as meetings stalling or making it difficult for some participants to speak up. Furthermore, they lacked the functionality to monitor meeting progress in real time and provide appropriate advice and suggestions, making it difficult to facilitate effective discussions. Furthermore, there were insufficient means to effectively utilize past meeting data, which often led to repeated discussions of the same issues.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1177] In this invention, the server includes: means for receiving voice data and converting it into text data using an automatic speech recognition engine; means for analyzing it using a natural language processing engine; means for identifying the user's emotions based on the analysis results, monitoring the progress of the meeting, and detecting abnormal progress; means for generating advice and suggestions based on the detection results and the identified emotions and transmitting the suggestions to the terminal; and means for analyzing past meeting data, generating a meeting playbook using a generative AI model, and transmitting the playbook to the terminal. This enables real-time support for the progress of meetings and the provision of advice and suggestions that take participants' emotions into consideration. Furthermore, past meeting data can be utilized to generate and display a meeting playbook to guide effective discussions.

[1178] "Audio data" refers to recorded information of words or remarks made by users during a conference.

[1179] "Server" refers to a computer system that receives, analyzes, identifies emotions, monitors, generates advice, and generates meeting playbooks for voice data.

[1180] "Terminal" refers to a computer or smart device used by a user that collects voice data and displays advice, suggestions, and meeting playbooks sent from the server.

[1181] An "automatic speech recognition engine" refers to software technology for analyzing voice data and converting it into text data.

[1182] "Text data" refers to character information converted from voice data by an automatic voice recognition engine.

[1183] A "natural language processing engine" refers to a technology that analyzes text data, understands its content and context, and extracts information.

[1184] "Emotion engine" refers to software technology that identifies a user's emotions based on text data.

[1185] "Monitoring" refers to the process of observing the progress of a meeting in real time and detecting any anomalies or problems.

[1186] "Advice" refers to recommendations and suggestions generated by the server to improve the progress of the meeting and the quality of the discussion.

[1187] "Suggestion" refers to a specific solution or instruction for action provided by the server for a particular situation or problem.

[1188] A "generative AI model" refers to an artificial intelligence model that learns from past data and makes predictions and generates based on new data.

[1189] "Meeting playbooks" refer to templates and guidelines for conducting effective discussions, created by a generative AI model based on past meeting data.

[1190] "Real-time display" refers to the immediate display of received advice or suggestions on the device.

[1191] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, this system makes meetings more effective by combining it with an emotion engine that recognizes the user's emotions. The specific form of this system is shown below.

[1192] Overall system configuration

[1193] This system is roughly divided into three components: a server, a terminal, and a user. The server has an emotion engine built in.

[1194] Server Features

[1195] The server collects, analyzes, monitors, and generates suggestions for voice data. Specifically, the server receives voice data and converts it into text using an automatic speech recognition engine such as Google Cloud Speech-to-Text. It then analyzes the text data using a natural language processing engine such as GPT-4 to identify the topic and sentiment of the speech. It then uses an emotion engine such as IBM Watson Tone Analyzer to recognize the user's sentiment from the analysis results. This allows the server to monitor the progress of the meeting and detect any abnormalities.

[1196] If an anomaly is detected, the server generates advice or suggestions using a generative AI model (e.g., GPT-3) and sends them to the device in real time. The server also analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). This playbook provides templates to help facilitate effective discussions.

[1197] Device Features

[1198] The terminal is a computer or smart device used by the user that collects voice data and sends it to the server. The terminal also displays real-time advice and suggestions sent from the server and provides them to the user. It also receives and displays the generated meeting playbook and sentiment-related feedback. This can be achieved through a custom GUI in a web browser or a plugin for the meeting software (e.g., Zoom, Microsoft Teams).

[1199] User Roles

[1200] Users are participants in a meeting and generate voice data by speaking. This voice data is sent to the server via their devices. Users refer to the advice and suggestions sent from the server to guide the meeting appropriately. Users can also refer to the generated meeting playbook to receive support for effective question and answer sessions and discussions.

[1201] Specific examples

[1202] Example 1: Real-time feedback

[1203] User A says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using Google Cloud Speech-to-Text. Next, it analyzes the text data using the natural language processing engine GPT-4 and recognizes the speaker's emotions using IBM Watson Tone Analyzer. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server generates a suggestion such as "Before that, please ask for participant B's opinion," and sends feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who then uses them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[1204] Prompt Sentence Examples

[1205] "User A wants to move on to the next agenda item, but seems impatient. Suggest that they listen to Participant B's opinion."

[1206] Example 2: Using the Conference Playbook

[1207] The server analyzes past project management meeting data and learns patterns using a generative AI model (e.g., BERT). For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also analyzes the emotional trends of participants in past meetings using IBM Watson Tone Analyzer. The server then sends the generated playbook and advice based on the emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure smooth progress in meetings.

[1208] Prompt Sentence Examples

[1209] "This is a meeting playbook generated from past meeting data. Please ask questions and make suggestions to elicit opinions from your team members."

[1210] This system combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1211] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1212] Step 1: Collecting audio data

[1213] The device collects the voice data of the user during the conference. The voice data is recorded through the device's microphone. The input is the user's speech, and the output is the recorded voice data. Specifically, the conference application captures the voice data in real time and prepares it to be sent to the server.

[1214] Step 2: Sending the audio data to the server

[1215] The device sends the collected voice data to the server. The input is the recorded voice data, and the output is the voice data sent to the server. Specifically, the device's communication module is used to encrypt the voice data and send it to the server via the network.

[1216] Step 3: Converting audio data to text

[1217] The server converts the received voice data into text data using an automatic speech recognition engine such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data. Specifically, the automatic speech recognition engine extracts words and sentences from the voice and outputs them as structured text data.

[1218] Step 4: Analyzing the text data

[1219] The server analyzes the text data using a natural language processing engine such as GPT-4. The input is the text data, and the output is the analyzed text data and its topic and context information. Specifically, the natural language processing engine interprets the content of the text data, extracts important keywords and phrases, and classifies them according to topic and context.

[1220] Step 5: Identify the emotion

[1221] The server uses an emotion engine such as IBM Watson Tone Analyzer to identify the user's emotion from the analyzed text data. The input is the analyzed text data, and the output is the identified emotion information. Specifically, the emotion engine analyzes the wording and structure in the text to identify the user's emotional state (e.g., joy, sadness, anger).

[1222] Step 6: Monitor the progress of the meeting

[1223] The server monitors the progress of the meeting in real time based on the identified emotions and text data. The input is emotion information and analyzed text data, and the output is the current progress of the meeting and the detection results of abnormal progress. Specifically, the server tracks this data and analyzes the frequency and content of comments, participants' reactions, etc.

[1224] Step 7: Generate advice and suggestions

[1225] The server generates advice or suggestions using a generative AI model (e.g., GPT-3) based on the detection results and identified emotions. The input is the abnormal progress detection results and emotion information, and the output is text data of the advice or suggestions. Specifically, the server forms specific suggestions such as "In this situation, please say this next" based on the generative AI model.

[1226] Step 8: Submit and view tips and suggestions

[1227] The server sends the generated advice or suggestion to the terminal, which displays it in real time. The input is text data of the advice or suggestion, and the output is the advice or suggestion displayed on the terminal. Specifically, it is displayed to the user as a pop-up window or a notification message using the terminal's display device.

[1228] Step 9: Analyze historical data and generate meeting playbooks

[1229] The server analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). The input is past meeting data, and the output is the generated meeting playbook. Specifically, the generative AI model learns from the data, extracts patterns of successful discussions and effective questions, and compiles them into templates.

[1230] Step 10: Submit and view the meeting playbook

[1231] The server sends the generated conference playbook to the terminal, which receives and displays it. The input is the generated conference playbook, and the output is the conference playbook displayed on the terminal. Specifically, it is provided to the user as a template or guideline displayed on the GUI.

[1232] (Application example 2)

[1233] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] Current food delivery services lack a system that can quickly respond to the difficulties and emotional changes that delivery workers face in the field. This can easily lead to delivery delays and reduced customer satisfaction. To provide an environment where delivery workers can perform their work efficiently and with peace of mind, a system is needed that monitors the progress of deliveries in real time and provides appropriate support and suggestions.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1236] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data received by the server into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of delivery based on the results of the analysis by the server and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the advice or suggestions received in real time, means for the server to analyze past delivery data and generate a delivery playbook using generative artificial intelligence and transmit it to the terminal, and means for the terminal to display the delivery playbook received. This enables delivery personnel to respond quickly and appropriately at the scene.

[1237] "Voice data" is information collected in digital form from the voices of delivery personnel.

[1238] "Real-time" means that data is processed immediately, without delay.

[1239] The "server" is a central system that receives and analyzes voice data and sends appropriate suggestions and feedback to the device.

[1240] An "automatic speech recognition engine" is a technology for converting voice data into text data.

[1241] "Text data" is character information converted by a voice recognition engine.

[1242] A "natural language processing engine" is a technology for analyzing text data and identifying meaning and emotion.

[1243] "Progress" refers to the current status in the delivery process.

[1244] "Monitoring" means watching progress in real time.

[1245] "Abnormal" refers to a situation where delivery is not proceeding as planned.

[1246] "Advice" is a suggestion of specific action to the delivery person.

[1247] A "suggestion" is advice for the delivery person to improve the current situation.

[1248] A "terminal" is a receiving device carried by a delivery person.

[1249] "Historical delivery data" is a collection of information about deliveries that have been made previously.

[1250] "Generative artificial intelligence" is a technology for generating new insights and suggestions from data.

[1251] The "Delivery Playbook" is a guidebook created based on past delivery data to support efficient delivery.

[1252] "Display" means providing information visually on a terminal.

[1253] This invention relates to a system that supports food delivery operations. In particular, it is a system for realizing a smart device application that recognizes the emotions and situations of delivery personnel in real time and provides appropriate advice and suggestions. This system is broadly composed of three elements: a server, a terminal, and a delivery personnel. The specific form of the system is shown below.

[1254] Overall system configuration

[1255] This system sends the delivery person's voice data to a server in real time, where it is converted into text data using an automatic speech recognition engine. The voice data is then analyzed using a natural language processing engine to monitor the progress of the delivery. If an abnormality is detected, the server generates appropriate advice or suggestions and sends them to the terminal. The terminal displays these advice and suggestions in real time, helping the delivery person quickly choose their next action. The server also analyzes past delivery data and uses generative artificial intelligence to generate a delivery playbook, which is sent to the terminal. The delivery playbook serves as a guideline for efficiently carrying out delivery work.

[1256] Hardware and software used

[1257] Hardware

[1258] Smartphone: A device carried by delivery personnel that collects voice data and displays advice and suggestions.

[1259] Server: A central system that analyzes data, generates proposals, and analyzes historical data.

[1260] software

[1261] Automatic speech recognition engine: Converts voice data into text data (e.g., Google Cloud Speech-to-Text).

[1262] Natural language processing engines: Analyze text data and identify meaning and sentiment (e.g., spaCy, NLTK).

[1263] Emotion Recognition AI: Recognize emotions from text data (e.g., Microsoft Azure Emotion API).

[1264] Generative AI: Generate new insights and suggestions from data (e.g., OpenAI GPT-4).

[1265] Program processing

[1266] Server processing

[1267] The server receives voice data sent by the delivery person. It then uses an automatic speech recognition engine to convert the voice data into text data. The converted text data is analyzed by a natural language processing engine to identify the content and emotion of the speech. Emotion recognition AI is used to recognize the emotions of the delivery person and monitor the delivery progress. If abnormal progress is detected, the server uses generation AI to generate appropriate advice or suggestions and send them to the device.

[1268] Specific example explanation

[1269] Example 1: Real-time support

[1270] When a delivery person says, "I'm going to be late," this voice data is sent from the smartphone to a server. The server receives the voice data and converts it into text using Google Cloud Speech-to-Text. The text data is analyzed using spaCy, and the keyword "I'm going to be late" is extracted. When analyzed using the Microsoft Azure Emotion API, it is identified that the delivery person is feeling anxious. The suggestion generated using OpenAI GPT-4 is the message, "Please check if there is a detour nearby. Also, if congestion persists, we recommend that you report the delay to customers and take a break." This message is immediately displayed on the delivery person's smartphone. Below is an example of a prompt.

[1271] Example prompt: What is the next action the delivery person should take if "It looks like it's going to be late" and "The roads are busy"? Generate a feedback statement.

[1272] In this way, delivery personnel can respond quickly and appropriately on site.

[1273] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1274] Step 1:

[1275] During delivery, the user inputs voice data into the smartphone, for example, "I'm going to be late, the roads are congested." This voice data is collected and sent from the smartphone to the server in real time.

[1276] Input: Delivery person's voice data

[1277] Output: Audio data sent to the server

[1278] Step 2:

[1279] The server converts the received voice data into text data using an automatic speech recognition engine (Google Cloud Speech-to-Text). Through this process, the voice data is converted into text information.

[1280] Input: Audio data

[1281] Output: Text data

[1282] Step 3:

[1283] The server analyzes the obtained text data using a natural language processing engine (spaCy or NLTK). Keywords in the text data (e.g., "I'm going to be late" or "The roads are congested") are extracted, and the topic and content of the utterance are identified.

[1284] Input: Text data

[1285] Output: Parsed topic and content information

[1286] Step 4:

[1287] The server uses emotion recognition AI (Microsoft Azure Emotion API) to recognize the emotions of the delivery person from the analysis results. Emotion recognition determines whether the delivery person is feeling anxious or stressed from the text data.

[1288] Input: Parsed topic and content information

[1289] Output: Recognized emotion data

[1290] Step 5:

[1291] The server monitors the delivery progress based on the analysis results and sentiment data, and if an abnormal progress (e.g., risk of delay) is detected, it proceeds to the next step.

[1292] Input: Recognized emotion data

[1293] Output: Progress determination result

[1294] Step 6:

[1295] The server uses a generative AI model (OpenAI GPT-4) to generate appropriate advice and suggestions. The prompt sentence is set as "What is the next action the delivery person should take if the delivery person is likely to be late and the roads are congested?" and the generated suggestions are created.

[1296] Input: Progress status determination result

[1297] Output: Generated advice and suggestions

[1298] Step 7:

[1299] The server sends the generated advice and suggestions to the terminal.

[1300] Input: Generated advice and suggestions

[1301] Output: Tips and suggestions sent to the terminal

[1302] Step 8:

[1303] The device displays the received advice and suggestions in real time. The delivery person can then refer to these advice and suggestions and select the appropriate action to carry out their task. Specifically, they can check for detours according to the example sentences displayed, or report delays to the customer.

[1304] Input: Tips and suggestions sent to your device

[1305] Output: Tips and suggestions displayed on the terminal

[1306] This allows delivery personnel to respond quickly and appropriately on site.

[1307] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1308] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1309] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1310] [Fourth embodiment]

[1311] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1312] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1313] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1314] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1315] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1316] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1317] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1318] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1319] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1320] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1321] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1322] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1323] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1324] The present invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. The specific form of this system is shown below.

[1325] Overall system configuration

[1326] This system is broadly composed of three elements: a server, a terminal, and a user.

[1327] server

[1328] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and sentiment of the speech. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates advice and suggestions based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[1329] Terminal

[1330] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. It also receives and displays the generated meeting playbook for the user to refer to.

[1331] User

[1332] Users are participants in a conference and make comments during the conference. The voice data of the user's speech is sent to the server via the terminal, where it is analyzed. The user can refer to the advice and suggestions displayed on the terminal to make comments at the appropriate time and to proceed with the conference.

[1333] Program processing

[1334] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and sentiment of the comments. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[1335] Specific examples

[1336] Example 1: Real-time feedback

[1337] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the progress is monitored. If a topic that requires discussion has been skipped, the server generates a suggestion such as "Before moving on, please ask participant B for his opinion," and sends it to the device. The device receives this suggestion and displays it to the user, who then proceeds with the meeting by saying, "Before that, Mr. B, do you have any comments?"

[1338] Example 2: Using the Conference Playbook

[1339] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" The server sends the generated playbook to the device, which receives it and displays it on a GUI. Users can refer to the playbook to ask effective questions and keep the meeting running smoothly.

[1340] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1341] The processing flow will be explained below.

[1342] Step 1:

[1343] A user speaks during a conference, and audio data is collected by the device's microphone.

[1344] Step 2:

[1345] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[1346] Step 3:

[1347] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[1348] Step 4:

[1349] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to determine the topic, sentiment, and importance of the utterance.

[1350] Step 5:

[1351] The server monitors the progress of the meeting based on the analysis results, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[1352] Step 6:

[1353] If the server detects abnormal progress, it generates appropriate advice or suggestions, such as "You might want to seek further feedback on this topic."

[1354] Step 7:

[1355] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[1356] Step 8:

[1357] The advice and suggestions received by the device are displayed in a format that provides a good user experience (UX), presented through a GUI (Graphical User Interface).

[1358] Step 9:

[1359] Users can check the advice and suggestions displayed on their device screen and proceed with the meeting based on them, for example, by asking specific participants for their opinions or changing the focus of the discussion.

[1360] Step 10:

[1361] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[1362] Step 11:

[1363] The server generates a meeting playbook and sends it to the device, making it available before and during the meeting.

[1364] Step 12:

[1365] The terminal displays the received meeting playbook, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[1366] The process flow of this system will help meetings run smoothly and improve the quality of discussions, allowing all participants to participate effectively, greatly increasing meeting efficiency.

[1367] Example 1

[1368] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1369] Improving the progress of meetings and the quality of discussions is a crucial issue in modern organizations. However, meetings often do not proceed as planned, discussions go off track, and some participants find it difficult to speak up. Furthermore, there is a lack of methods to effectively utilize past meeting data to conduct meetings effectively. To solve these issues, real-time progress management and appropriate feedback to participants are required.

[1370] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1371] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the received voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the conference based on the analysis results and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the received advice or suggestions in real time, means for the server to analyze past conference data and generate a conference playbook using artificial intelligence and transmit it to the terminal, and means for the terminal to display the received conference playbook. This makes it possible to manage the progress of the conference in real time and provide appropriate advice immediately when an abnormal situation is detected. Furthermore, past conference data can be used to generate effective playbooks, improving the quality and efficiency of conferences.

[1372] "Voice data" refers to data for recording and transmitting user speech and voice information in digital format.

[1373] "Server" means a computer system that receives, analyzes, monitors, generates, and transmits audio data.

[1374] A "terminal" is a computer or smart device used by a user, and is a device for collecting voice data and displaying advice and suggestions from the server.

[1375] A "user" is a participant in a conference, who speaks and receives advice and suggestions from the server.

[1376] An "automatic speech recognition engine" is a software component for converting voice data into text data.

[1377] A "natural language processing engine" is a software component that analyzes text data and identifies meaning and sentiment.

[1378] "Monitoring" is the act or process of watching the progress of a meeting in real time and detecting anomalies.

[1379] An "abnormal progress" is when a meeting deviates from the planned course of action, certain topics are skipped, or necessary discussions are not held.

[1380] "Advice" refers to instructions or suggestions provided to facilitate the smooth running of a meeting.

[1381] A "proposal" is a specific action or instruction to solve a problem.

[1382] A "meeting playbook" is a document generated based on past meeting data that includes materials and sample questions for conducting effective meetings.

[1383] "Real-time" refers to processing or communication that occurs nearly immediately or without delay.

[1384] "Generative AI" is an artificial intelligence technology that analyzes large amounts of data, learns patterns and knowledge, and generates new information and suggestions.

[1385] System configuration

[1386] The conference support system of the present invention is composed of three main elements: a server, a terminal, and a user.

[1387] server

[1388] The server is responsible for receiving, analyzing, and monitoring the audio data, and generating advice and suggestions. Specifically, the process proceeds as follows:

[1389] Receiving and converting audio data

[1390] The server receives voice data from the device, converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API), and sends the text data to a natural language processing engine (e.g., OpenAI's GPT-3) for analysis.

[1391] Text data analysis

[1392] The server then analyzes the converted text data using a natural language processing engine to identify the topic and sentiment of the comments, and monitors the progress of the meeting in real time based on the analysis results to detect any anomalies.

[1393] Generating advice and suggestions

[1394] If the server detects an abnormality in the progress, it generates appropriate advice or suggestions. For example, it may generate a specific suggestion such as, "Please ask participant B for his / her opinion before moving on to the next agenda item." This is generated using a generative AI model (e.g., GPT-3).

[1395] Generate a conference playbook

[1396] The server analyzes past meeting data and generates a meeting playbook to support effective progress. The playbook is a document that learns and summarizes past patterns and effective questions.

[1397] Terminal

[1398] The device collects user utterances, transmits the audio data to the server, and displays advice, suggestions, and meeting playbooks received from the server in real time.

[1399] Collection and transmission of voice data

[1400] The device collects the user's speech using a microphone and transmits it to the server in real time using WebSocket or HTTP POST requests.

[1401] View advice, suggestions and playbooks

[1402] Advice and suggestions received from the server are displayed in real time on the terminal, possibly as a pop-up window or in a notification area. The meeting playbook sent from the server is also displayed on the GUI for the user to refer to in real time.

[1403] User

[1404] Users are participants in the conference and make comments. Their comments are collected by their devices and sent to the server. The progress of the conference is adjusted based on advice and suggestions displayed on the devices.

[1405] User comments and reviews

[1406] The user makes a statement and reviews the advice and suggestions displayed on the device, which determines what to say next and how to proceed.

[1407] Specific examples

[1408] Real-time feedback

[1409] 1. A user says, "Let's move on to the next item on the agenda."

[1410] 2. The device collects the voice data and sends it to the server.

[1411] 3. The server converts the voice data into text data using the Google Speech-to-Text API.

[1412] 4. The server analyzes the text data using OpenAI's GPT-3 and monitors progress.

[1413] 5. The server determines that a topic requiring discussion has been skipped and generates an advice.

[1414] 6. The server sends the advice to the terminal: "Please ask participant B for his opinion before moving on."

[1415] 7. The device receives the advice and displays it to the user in a pop-up notification.

[1416] 8. The user says, "Before that, Mr. B, do you have any comments?"

[1417] Using the Conference Playbook

[1418] 1. The server analyzes past meeting data and generates a meeting playbook containing effective questions.

[1419] 2. The server sends the generated conference playbook to the terminal.

[1420] 3. The terminal receives the conference playbook and displays it on the GUI.

[1421] 4. The user consults the meeting playbook and schedules the next question to be asked.

[1422] 5. Users effectively facilitate meetings by asking the question, "What's the progress on risk management?"

[1423] Prompt Sentence Examples

[1424] "Generate a reminder to avoid skipping topics that need discussion after saying let's move on to the next agenda item."

[1425] "Generate a playbook with useful questions for project management meetings."

[1426] In this way, the system of the present invention can support the progress of meetings and improve the quality of discussions by providing real-time feedback and utilizing past data.

[1427] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1428] System program processing flow

[1429] Server Processing Steps

[1430] Step 1: Receiving audio data

[1431] Step 2: Convert the audio data

[1432] Step 3: Analyzing the text data

[1433] Step 4: Monitor the progress of the meeting

[1434] Step 5: Generate advice and suggestions

[1435] Step 6: Submit your advice and suggestions

[1436] Step 7: Generate the conference playbook

[1437] Terminal processing steps

[1438] Step 1: Collecting audio data

[1439] Step 2: Sending audio data

[1440] Step 3: Receiving advice and suggestions

[1441] Step 4: Viewing Advice and Suggestions

[1442] Step 5: Receive the conference playbook

[1443] Step 6: View the conference playbook

[1444] User processing steps

[1445] Step 1: Speak up

[1446] Step 2: Review advice and suggestions

[1447] Step 3: Refer to the conference playbook

[1448] Detailed explanation of the processing steps

[1449] Server Processing Steps

[1450] Step 1: Receiving audio data

[1451] The server receives audio data from the device. The input is the audio data sent from the device, and the output is the received audio data. This audio data is received via WebSocket or HTTP POST request.

[1452] Step 2: Convert the audio data

[1453] The server converts the received voice data into text data using an automatic speech recognition engine (e.g., Google Speech-to-Text API). The input is the received voice data, and the output is the converted text data. Converting the voice data into text data prepares it for natural language processing.

[1454] Step 3: Analyzing the text data

[1455] The server analyzes the converted text data using a natural language processing engine (e.g., OpenAI's GPT-3). The input is text data, and the output is the analysis results. The analysis results include the topic and sentiment of the comments. This analysis allows for a concrete understanding of the progress of the meeting.

[1456] Step 4: Monitor the progress of the meeting

[1457] The server monitors the progress of the conference based on the results of analyzing the text data. The input is the analysis results, and the output is monitoring information. This information is used to detect abnormal progress (such as skipping topics or delays in progress).

[1458] Step 5: Generate advice and suggestions

[1459] If the server detects abnormal progress, it generates advice or suggestions. The input is monitoring information, and the output is the generated advice or suggestions. A generative AI model (e.g., GPT-3) is used to suggest specific actions. For example, it generates suggestions such as, "Please ask participant B for his opinion before moving on to the next agenda item."

[1460] Step 6: Submit your advice and suggestions

[1461] The server sends generated advice and suggestions to the device. The input is the generated advice and suggestions, and the output is the sent advice and suggestions. The low-latency WebSocket protocol is used to notify the device in real time.

[1462] Step 7: Generate the conference playbook

[1463] The server analyzes past meeting data and generates a meeting playbook. The input is past meeting data, and the output is the generated meeting playbook. A generative AI model is used to learn past patterns and effective example questions, and generate a specific playbook.

[1464] Terminal processing steps

[1465] Step 1: Collecting audio data

[1466] The device collects the user's speech from a microphone. The input is the user's speech, and the output is the collected voice data. This voice data is sent to the server in real time.

[1467] Step 2: Sending audio data

[1468] The device sends collected audio data to the server. The input is the collected audio data, and the output is the audio data sent to the server. Data is sent with low latency using WebSocket or HTTP POST requests.

[1469] Step 3: Receiving advice and suggestions

[1470] The device receives advice and suggestions sent from the server. The input is the advice and suggestions sent from the server, and the output is the received advice and suggestions. The advice and suggestions are received in real time and reflected in the next action.

[1471] Step 4: Viewing Advice and Suggestions

[1472] The advice and suggestions received by the device are displayed in real time on the GUI. The input is the received advice and suggestions, and the output is the displayed advice and suggestions. The user is notified using a pop-up window or notification area.

[1473] Step 5: Receive the conference playbook

[1474] The terminal receives the meeting playbook sent from the server. The input is the meeting playbook sent from the server, and the output is the received meeting playbook. It is received in real time and used during the meeting.

[1475] Step 6: View the conference playbook

[1476] The terminal displays the received conference playbook on the GUI. The input is the received conference playbook, and the output is the displayed conference playbook, allowing the user to refer to it immediately.

[1477] User processing steps

[1478] Step 1: Speak up

[1479] Users make comments during a conference. The input is the progress of the conference, and the output is the user's comments. These comments are collected by the terminal and sent to the server.

[1480] Step 2: Review advice and suggestions

[1481] The user reviews the advice and suggestions displayed on the device. The input is the displayed advice and suggestions, and the output is the user's review. The next action is determined based on the review result.

[1482] Step 3: Refer to the conference playbook

[1483] The user refers to the meeting playbook received from the server. The input is the displayed meeting playbook, and the output is the next action based on the reference. The playbook is used to adjust the content and timing of remarks.

[1484] In this way, the elements of the server, terminals, and users work together to realize a system that improves the progress of meetings and the quality of discussions.

[1485] (Application example 1)

[1486] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1487] Conventional meeting support systems lack effective means for improving meeting progress and the quality of discussions. In particular, in production meetings in factories and other situations where real-time feedback is required, efficient and prompt progress is essential, but current systems are insufficient. Furthermore, because they are unable to perform sentiment analysis during ongoing meetings, it is difficult to accurately detect problems that arise during meetings and provide advice and suggestions for improvement. Therefore, a new system is needed to improve the efficiency and quality of meetings.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1489] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of the meeting based on the results and detecting abnormal progress, means for generating advice and suggestions and transmitting them to the terminal, means for identifying emotions and generating appropriate advice and suggestions based thereon and transmitting them to the terminal, means for generating a meeting playbook and transmitting it to the terminal, and means for displaying the advice and suggestions based on the emotions received by the terminal in real time. This makes it possible to provide appropriate feedback based on emotions in production meetings and other meetings and effectively support the progress of the meeting.

[1490] "Audio data" refers to data in which the sound of a meeting, conversation, etc. is recorded in digital format.

[1491] "Server" is a computer system for collecting, analyzing, monitoring, and generating suggestions from speech data.

[1492] An "automatic speech recognition engine" is a technology that converts voice data into text data.

[1493] A "natural language processing engine" is a technology for analyzing text data and understanding its content.

[1494] "Emotion identification" is a technology that determines the emotion of a speaker based on text data.

[1495] "Real-time" refers to processing occurring immediately without delay.

[1496] "Progress monitoring" is a technique for constantly monitoring and recording the progress of meetings and work.

[1497] "Anomaly detection" refers to discovering when progress is not normal.

[1498] "Advice and suggestion generation" is a technology that generates advice and suggestions to support the progress of meetings and work based on the analysis results.

[1499] A "meeting playbook" is a guide document generated by analyzing past meeting data to support the progress of the next meeting.

[1500] A "terminal" is a device that receives advice, suggestions, and conference playbooks sent from the server and displays them to the user.

[1501] The system of the present invention is characterized by collecting voice data in real time and analyzing it on a server in order to improve the progress of meetings and the quality of discussions. Furthermore, advice and suggestions are generated based on the analysis results, and are sent to and displayed on terminals, thereby supporting efficient progress of meetings.

[1502] Overall system configuration

[1503] This system is broadly composed of three elements: a server, a terminal, and a user.

[1504] server

[1505] The server collects, analyzes, monitors, and generates suggestions from audio data. It uses the following hardware and software:

[1506] Hardware: A server computer equipped with a high-performance processor

[1507] Software: Google Cloud Speech-to-Text API, automatic speech recognition engine, TextBlob, natural language processing engine

[1508] The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed using a natural language processing engine to identify the topic and sentiment of the speech. This allows the progress of the meeting to be monitored and abnormal progress to be detected. For example, when a speech such as "Let's move on to the next agenda item" is heard using text analysis, the system checks based on this content to see if necessary topics have been skipped. If necessary, it generates a suggestion such as "Please ask participants for their opinions before moving on."

[1509] Terminal

[1510] A terminal is a computer or smart device used by a user that has the following functions:

[1511] Audio data collection: Use a microphone to collect meeting audio

[1512] Data transmission: Collected voice data is sent to the server in real time.

[1513] Data display: Real-time display of advice, suggestions, and meeting playbooks sent from the server

[1514] The terminal receives suggestions from the server and displays them to the user, for example, "Please review your risk management before proceeding to the next step" based on the meeting playbook.

[1515] User

[1516] Users are participants in a conference and make comments during the conference. The voice data spoken by the user is sent to the server via the terminal. The results of analysis by the server are displayed on the terminal, and the user can use this information to proceed with the conference. For example, by saying, "Before we begin, do you have any comments?", the ongoing discussion can be stimulated.

[1517] Specific examples

[1518] When a user says, "There is a delay in the next production line," the voice data is sent to the server via the device. The server converts the voice data into text data using an automatic speech recognition engine and analyzes it using TextBlob. Based on the analysis results, the server generates a suggestion saying, "There appears to be a delay. Let's identify the specific problem," and sends it to the device. The device receives this suggestion and displays it to the user.

[1519] Prompt Sentence Examples

[1520] Input: "The next production line is experiencing delays."

[1521] output:

[1522] Voice recognition result: There is a delay in the next production line

[1523] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[1524] Tip: It looks like you're having a problem. Let's find out what the problem is.

[1525] In this way, the system of the present invention supports the progress of meetings and improves the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1526] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1527] Step 1:

[1528] This system collects voice data spoken by users during a conference. The input is the voice spoken by users during a conference. The device collects this voice using a built-in microphone or an external microphone and records it as digital voice data. The output is the collected digital voice data.

[1529] Step 2:

[1530] The collected voice data is sent to the server in real time. The input is the digital voice data collected in step 1. The device sends this voice data to the server via the network. The output is the voice data received by the server.

[1531] Step 3:

[1532] The server converts the received voice data into text data using an automatic speech recognition engine. The input is the voice data received by the server. The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The output is the converted text data.

[1533] Step 4:

[1534] The server analyzes the text data using a natural language processing engine to identify the topic and sentiment of the utterance. The input is the text data generated in step 3. The server analyzes the text data using TextBlob to identify the topic and sentiment. Specifically, it extracts important keywords from the text and performs sentiment analysis. The output is the topic and sentiment data as the analysis results.

[1535] Step 5:

[1536] The server monitors the progress of the meeting based on the analysis results and detects abnormal progress. The input is the topic and emotion data obtained in step 4. The server monitors the progress in real time and detects abnormal situations, such as when an important topic in the meeting is skipped. The output is the detection result when an abnormality is detected.

[1537] Step 6:

[1538] The server generates advice and suggestions based on the detection results and sends them to the terminal. The input is the anomaly detection result from step 5. The server generates advice and suggestions for the next step based on the analysis results and detection results and sends these advice and suggestions to the terminal. The output is the generated advice and suggestions.

[1539] Step 7:

[1540] The device displays the received advice or suggestion to the user in real time. The input is the advice or suggestion generated in step 6. The device displays this advice or suggestion on the screen and notifies the user. The output is the advice or suggestion that the user can see on the screen.

[1541] Step 8:

[1542] The server analyzes past meeting data and uses generative AI to generate a meeting playbook, which is then sent to the device. The input is the previously accumulated meeting data. The server uses a generative AI model to learn past patterns and effective questions, and generates the meeting playbook. The output is the generated meeting playbook.

[1543] Step 9:

[1544] The terminal displays the received conference playbook. The input is the conference playbook generated in step 8. The terminal displays the generated playbook on the screen so that the user can refer to it. The output is a conference playbook that the user can check.

[1545] Examples of prompt statements

[1546] Input: "The next production line is experiencing delays."

[1547] output:

[1548] Voice recognition result: There is a delay in the next production line

[1549] Analysis result: {'topic': 'production meeting', 'sentiment': 'negative'}

[1550] Tip: It looks like you're having a problem. Let's find out what the problem is.

[1551] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1552] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, by combining it with an emotion engine that recognizes the user's emotions, the progress of meetings can be made more effective. Specific examples of this system are shown below.

[1553] Overall system configuration

[1554] This system is broadly composed of three elements: the server, the terminal, and the user. The server also has an emotion engine built in.

[1555] server

[1556] The server collects, analyzes, monitors, and generates suggestions for voice data. It receives voice data and converts it into text data using an automatic speech recognition engine. It then analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. It then uses an emotion engine to recognize the user's emotions from the analysis results. This allows it to monitor the progress of the meeting and detect abnormal progress. The server also generates appropriate advice and suggestions that take emotions into account based on the detection results and sends them to the terminal in real time. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[1557] Terminal

[1558] The terminal is a computer or smart device used by the user, which collects voice data and sends it to the server. The terminal displays advice and suggestions sent from the server in real time and presents them to the user. The terminal also receives and displays the generated meeting playbook and emotion-related feedback for the user to refer to.

[1559] User

[1560] Users are participants in a conference and speak during the conference. The voice data they speak is sent to the server via their device, where it is analyzed. They use the advice, suggestions, and emotion-related feedback displayed on their device to speak at the appropriate time and guide the conference.

[1561] Program processing

[1562] The server-side program performs processing in the following steps: First, it receives voice data and converts it into text data using an automatic speech recognition engine. Next, it analyzes the text data using a natural language processing engine to identify the topic and emotion of the speech. Finally, it uses an emotion engine to recognize the user's emotion from the analysis results. It then monitors the progress of the meeting and generates advice and suggestions if abnormal progress is detected. These advice and suggestions are sent to the terminal in real time, and emotional feedback is also provided. It also analyzes past meeting data and generates a meeting playbook using generative artificial intelligence.

[1563] Specific examples

[1564] Example 1: Real-time feedback

[1565] The user says, "Let's move on to the next item on the agenda." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine, and the emotion engine recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for Participant B's opinion," and send feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who can then use them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[1566] Example 2: Using the Conference Playbook

[1567] The server analyzes past project management meeting data and uses generative AI to learn patterns. For example, it generates a meeting playbook that includes the following questions: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also uses an emotion engine to analyze the emotional tendencies of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure the meeting proceeds smoothly.

[1568] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1569] The processing flow will be explained below.

[1570] Step 1:

[1571] A user speaks during a conference, and audio data is collected by the device's microphone.

[1572] Step 2:

[1573] The device transmits collected audio data to the server in real time using a secure communication protocol (e.g., HTTPS or WebSocket).

[1574] Step 3:

[1575] The server passes the received voice data to an automatic speech recognition (ASR) engine, which converts the voice data into text data.

[1576] Step 4:

[1577] The server passes the converted text data to a natural language processing (NLP) engine, which analyzes the text data to identify the topic and importance of the comment.

[1578] Step 5:

[1579] The server passes the analysis results obtained by the NLP engine to the emotion engine, which recognizes the speaker's emotions. The emotion engine evaluates the type of emotion (e.g., joy, anger, anxiety, etc.) and intensity of the emotion contained in the speech.

[1580] Step 6:

[1581] The server monitors the progress of the meeting based on the analysis results and emotion data, detecting topics where progress is stalled or situations where a particular participant is not speaking enough.

[1582] Step 7:

[1583] The server generates appropriate advice and suggestions based on the abnormal progress and emotion data, such as "You might want to ask for more opinions on this topic" or "Participant C is showing anxiety, so ask for their opinion."

[1584] Step 8:

[1585] The server generates advice and suggestions and sends them to the terminal in real time, allowing immediate follow-up of the progress of the meeting.

[1586] Step 9:

[1587] The device displays the received advice, suggestions, and emotion-related feedback to the user through a GUI (Graphical User Interface).

[1588] Step 10:

[1589] Users can view advice, suggestions, and emotional feedback displayed on their device screen and use it to guide the meeting, for example by asking specific participants for their opinions or changing the focus of the discussion.

[1590] Step 11:

[1591] The server collects and analyzes past meeting data, and uses generative AI to learn patterns and key points to generate a "meeting playbook."

[1592] Step 12:

[1593] The server sends the generated meeting playbook and the results of sentiment analysis to the device, making them available before and during the meeting.

[1594] Step 13:

[1595] The device displays the received meeting playbook and sentiment analysis results, which users can refer to as reference material to improve the progress of the meeting and the quality of the discussion.

[1596] Specific examples

[1597] Example 1: Real-time feedback

[1598] The user says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using an automatic speech recognition engine. The text data is then analyzed by a natural language processing engine. The emotion engine then recognizes the speaker's emotions. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server will generate a suggestion such as "Before that, please ask for participant B's opinion," and send it to the device in real time. The device displays this, and the user can respond to the suggestion by saying, "Before that, Mr. B, do you have any comments?", thereby ensuring the smooth progress of the meeting.

[1599] Example 2: Using the Conference Playbook

[1600] The server analyzes past meeting data and learns patterns using generative artificial intelligence. For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions were effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It then uses an emotion engine to analyze the emotional trends of participants in past meetings. The server then sends the generated playbook and advice based on emotion analysis to the device, which receives it and displays it on a GUI. The user can use this as a reference to ask effective questions and keep the meeting running smoothly.

[1601] In this way, the system of the present invention combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1602] Example 2

[1603] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1604] Conventional meeting support systems did not take into account the emotions of participants, which resulted in problems such as meetings stalling or making it difficult for some participants to speak up. Furthermore, they lacked the functionality to monitor meeting progress in real time and provide appropriate advice and suggestions, making it difficult to facilitate effective discussions. Furthermore, there were insufficient means to effectively utilize past meeting data, which often led to repeated discussions of the same issues.

[1605] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1606] In this invention, the server includes: means for receiving voice data and converting it into text data using an automatic speech recognition engine; means for analyzing it using a natural language processing engine; means for identifying the user's emotions based on the analysis results, monitoring the progress of the meeting, and detecting abnormal progress; means for generating advice and suggestions based on the detection results and the identified emotions and transmitting the suggestions to the terminal; and means for analyzing past meeting data, generating a meeting playbook using a generative AI model, and transmitting the playbook to the terminal. This enables real-time support for the progress of meetings and the provision of advice and suggestions that take participants' emotions into consideration. Furthermore, past meeting data can be utilized to generate and display a meeting playbook to guide effective discussions.

[1607] "Audio data" refers to recorded information of words or remarks made by users during a conference.

[1608] "Server" refers to a computer system that receives, analyzes, identifies emotions, monitors, generates advice, and generates meeting playbooks for voice data.

[1609] "Terminal" refers to a computer or smart device used by a user that collects voice data and displays advice, suggestions, and meeting playbooks sent from the server.

[1610] An "automatic speech recognition engine" refers to software technology for analyzing voice data and converting it into text data.

[1611] "Text data" refers to character information converted from voice data by an automatic voice recognition engine.

[1612] A "natural language processing engine" refers to a technology that analyzes text data, understands its content and context, and extracts information.

[1613] "Emotion engine" refers to software technology that identifies a user's emotions based on text data.

[1614] "Monitoring" refers to the process of observing the progress of a meeting in real time and detecting any anomalies or problems.

[1615] "Advice" refers to recommendations and suggestions generated by the server to improve the progress of the meeting and the quality of the discussion.

[1616] "Suggestion" refers to a specific solution or instruction for action provided by the server for a particular situation or problem.

[1617] A "generative AI model" refers to an artificial intelligence model that learns from past data and makes predictions and generates based on new data.

[1618] "Meeting playbooks" refer to templates and guidelines for conducting effective discussions, created by a generative AI model based on past meeting data.

[1619] "Real-time display" refers to the immediate display of received advice or suggestions on the device.

[1620] This invention relates to a support system that uses AI technology to improve the progress of meetings and the quality of discussions. In particular, this system makes meetings more effective by combining it with an emotion engine that recognizes the user's emotions. The specific form of this system is shown below.

[1621] Overall system configuration

[1622] This system is roughly divided into three components: a server, a terminal, and a user. The server has an emotion engine built in.

[1623] Server Features

[1624] The server collects, analyzes, monitors, and generates suggestions for voice data. Specifically, the server receives voice data and converts it into text using an automatic speech recognition engine such as Google Cloud Speech-to-Text. It then analyzes the text data using a natural language processing engine such as GPT-4 to identify the topic and sentiment of the speech. It then uses an emotion engine such as IBM Watson Tone Analyzer to recognize the user's sentiment from the analysis results. This allows the server to monitor the progress of the meeting and detect any abnormalities.

[1625] If an anomaly is detected, the server generates advice or suggestions using a generative AI model (e.g., GPT-3) and sends them to the device in real time. The server also analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). This playbook provides templates to help facilitate effective discussions.

[1626] Device Features

[1627] The terminal is a computer or smart device used by the user that collects voice data and sends it to the server. The terminal also displays real-time advice and suggestions sent from the server and provides them to the user. It also receives and displays the generated meeting playbook and sentiment-related feedback. This can be achieved through a custom GUI in a web browser or a plugin for the meeting software (e.g., Zoom, Microsoft Teams).

[1628] User Roles

[1629] Users are participants in a meeting and generate voice data by speaking. This voice data is sent to the server via their devices. Users refer to the advice and suggestions sent from the server to guide the meeting appropriately. Users can also refer to the generated meeting playbook to receive support for effective question and answer sessions and discussions.

[1630] Specific examples

[1631] Example 1: Real-time feedback

[1632] User A says, "Let's move on to the next agenda item." This voice data is sent to the server via the device. The server receives the voice data and converts it into text data using Google Cloud Speech-to-Text. Next, it analyzes the text data using the natural language processing engine GPT-4 and recognizes the speaker's emotions using IBM Watson Tone Analyzer. For example, if the speaker's emotions indicate impatience or dissatisfaction, the server generates a suggestion such as "Before that, please ask for participant B's opinion," and sends feedback that takes emotion into consideration to the device. The device receives these and displays them to the user, who then uses them to say, "Before that, Mr. B, do you have any comments?" and move the meeting forward.

[1633] Prompt Sentence Examples

[1634] "User A wants to move on to the next agenda item, but seems impatient. Suggest that they listen to Participant B's opinion."

[1635] Example 2: Using the Conference Playbook

[1636] The server analyzes past project management meeting data and learns patterns using a generative AI model (e.g., BERT). For example, it generates a meeting playbook with the following content: "In traditional project management meetings, the following three questions have been effective: 1. What is the progress on risk management? 2. What feedback have you received from team members?" It also analyzes the emotional trends of participants in past meetings using IBM Watson Tone Analyzer. The server then sends the generated playbook and advice based on the emotion analysis to the device, which receives it and displays it on a GUI. Users can use this information to ask effective questions and ensure smooth progress in meetings.

[1637] Prompt Sentence Examples

[1638] "This is a meeting playbook generated from past meeting data. Please ask questions and make suggestions to elicit opinions from your team members."

[1639] This system combines AI and an emotion engine to support the progress of meetings and improve the quality of discussions, thereby increasing the efficiency of meetings and enabling all participants to participate effectively.

[1640] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1641] Step 1: Collecting audio data

[1642] The device collects the voice data of the user during the conference. The voice data is recorded through the device's microphone. The input is the user's speech, and the output is the recorded voice data. Specifically, the conference application captures the voice data in real time and prepares it to be sent to the server.

[1643] Step 2: Sending the audio data to the server

[1644] The device sends the collected voice data to the server. The input is the recorded voice data, and the output is the voice data sent to the server. Specifically, the device's communication module is used to encrypt the voice data and send it to the server via the network.

[1645] Step 3: Converting audio data to text

[1646] The server converts the received voice data into text data using an automatic speech recognition engine such as Google Cloud Speech-to-Text. The input is voice data, and the output is the converted text data. Specifically, the automatic speech recognition engine extracts words and sentences from the voice and outputs them as structured text data.

[1647] Step 4: Analyzing the text data

[1648] The server analyzes the text data using a natural language processing engine such as GPT-4. The input is the text data, and the output is the analyzed text data and its topic and context information. Specifically, the natural language processing engine interprets the content of the text data, extracts important keywords and phrases, and classifies them according to topic and context.

[1649] Step 5: Identify the emotion

[1650] The server uses an emotion engine such as IBM Watson Tone Analyzer to identify the user's emotion from the analyzed text data. The input is the analyzed text data, and the output is the identified emotion information. Specifically, the emotion engine analyzes the wording and structure in the text to identify the user's emotional state (e.g., joy, sadness, anger).

[1651] Step 6: Monitor the progress of the meeting

[1652] The server monitors the progress of the meeting in real time based on the identified emotions and text data. The input is emotion information and analyzed text data, and the output is the current progress of the meeting and the detection results of abnormal progress. Specifically, the server tracks this data and analyzes the frequency and content of comments, participants' reactions, etc.

[1653] Step 7: Generate advice and suggestions

[1654] The server generates advice or suggestions using a generative AI model (e.g., GPT-3) based on the detection results and identified emotions. The input is the abnormal progress detection results and emotion information, and the output is text data of the advice or suggestions. Specifically, the server forms specific suggestions such as "In this situation, please say this next" based on the generative AI model.

[1655] Step 8: Submit and view tips and suggestions

[1656] The server sends the generated advice or suggestion to the terminal, which displays it in real time. The input is text data of the advice or suggestion, and the output is the advice or suggestion displayed on the terminal. Specifically, it is displayed to the user as a pop-up window or a notification message using the terminal's display device.

[1657] Step 9: Analyze historical data and generate meeting playbooks

[1658] The server analyzes past meeting data and generates a meeting playbook using a generative AI model (e.g., BERT). The input is past meeting data, and the output is the generated meeting playbook. Specifically, the generative AI model learns from the data, extracts patterns of successful discussions and effective questions, and compiles them into templates.

[1659] Step 10: Submit and view the meeting playbook

[1660] The server sends the generated conference playbook to the terminal, which receives and displays it. The input is the generated conference playbook, and the output is the conference playbook displayed on the terminal. Specifically, it is provided to the user as a template or guideline displayed on the GUI.

[1661] (Application example 2)

[1662] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1663] Current food delivery services lack a system that can quickly respond to the difficulties and emotional changes that delivery workers face in the field. This can easily lead to delivery delays and reduced customer satisfaction. To provide an environment where delivery workers can perform their work efficiently and with peace of mind, a system is needed that monitors the progress of deliveries in real time and provides appropriate support and suggestions.

[1664] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1665] In this invention, the server includes means for collecting voice data and transmitting it to the server in real time, means for converting the voice data received by the server into text data using an automatic speech recognition engine and analyzing it using a natural language processing engine, means for monitoring the progress of delivery based on the results of the analysis by the server and detecting abnormal progress, means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal, means for the terminal to display the advice or suggestions received in real time, means for the server to analyze past delivery data and generate a delivery playbook using generative artificial intelligence and transmit it to the terminal, and means for the terminal to display the delivery playbook received. This enables delivery personnel to respond quickly and appropriately at the scene.

[1666] "Voice data" is information collected in digital form from the voices of delivery personnel.

[1667] "Real-time" means that data is processed immediately, without delay.

[1668] The "server" is a central system that receives and analyzes voice data and sends appropriate suggestions and feedback to the device.

[1669] An "automatic speech recognition engine" is a technology for converting voice data into text data.

[1670] "Text data" is character information converted by a voice recognition engine.

[1671] A "natural language processing engine" is a technology for analyzing text data and identifying meaning and emotion.

[1672] "Progress" refers to the current status in the delivery process.

[1673] "Monitoring" means watching progress in real time.

[1674] "Abnormal" refers to a situation where delivery is not proceeding as planned.

[1675] "Advice" is a suggestion of specific action to the delivery person.

[1676] A "suggestion" is advice for the delivery person to improve the current situation.

[1677] A "terminal" is a receiving device carried by a delivery person.

[1678] "Historical delivery data" is a collection of information about deliveries that have been made previously.

[1679] "Generative artificial intelligence" is a technology for generating new insights and suggestions from data.

[1680] The "Delivery Playbook" is a guidebook created based on past delivery data to support efficient delivery.

[1681] "Display" means providing information visually on a terminal.

[1682] This invention relates to a system that supports food delivery operations. In particular, it is a system for realizing a smart device application that recognizes the emotions and situations of delivery personnel in real time and provides appropriate advice and suggestions. This system is broadly composed of three elements: a server, a terminal, and a delivery personnel. The specific form of the system is shown below.

[1683] Overall system configuration

[1684] This system sends the delivery person's voice data to a server in real time, where it is converted into text data using an automatic speech recognition engine. The voice data is then analyzed using a natural language processing engine to monitor the progress of the delivery. If an abnormality is detected, the server generates appropriate advice or suggestions and sends them to the terminal. The terminal displays these advice and suggestions in real time, helping the delivery person quickly choose their next action. The server also analyzes past delivery data and uses generative artificial intelligence to generate a delivery playbook, which is sent to the terminal. The delivery playbook serves as a guideline for efficiently carrying out delivery work.

[1685] Hardware and software used

[1686] Hardware

[1687] Smartphone: A device carried by delivery personnel that collects voice data and displays advice and suggestions.

[1688] Server: A central system that analyzes data, generates proposals, and analyzes historical data.

[1689] software

[1690] Automatic speech recognition engine: Converts voice data into text data (e.g., Google Cloud Speech-to-Text).

[1691] Natural language processing engines: Analyze text data and identify meaning and sentiment (e.g., spaCy, NLTK).

[1692] Emotion Recognition AI: Recognize emotions from text data (e.g., Microsoft Azure Emotion API).

[1693] Generative AI: Generate new insights and suggestions from data (e.g., OpenAI GPT-4).

[1694] Program processing

[1695] Server processing

[1696] The server receives voice data sent by the delivery person. It then uses an automatic speech recognition engine to convert the voice data into text data. The converted text data is analyzed by a natural language processing engine to identify the content and emotion of the speech. Emotion recognition AI is used to recognize the emotions of the delivery person and monitor the delivery progress. If abnormal progress is detected, the server uses generation AI to generate appropriate advice or suggestions and send them to the device.

[1697] Specific example explanation

[1698] Example 1: Real-time support

[1699] When a delivery person says, "I'm going to be late," this voice data is sent from the smartphone to a server. The server receives the voice data and converts it into text using Google Cloud Speech-to-Text. The text data is analyzed using spaCy, and the keyword "I'm going to be late" is extracted. When analyzed using the Microsoft Azure Emotion API, it is identified that the delivery person is feeling anxious. The suggestion generated using OpenAI GPT-4 is the message, "Please check if there is a detour nearby. Also, if congestion persists, we recommend that you report the delay to customers and take a break." This message is immediately displayed on the delivery person's smartphone. Below is an example of a prompt.

[1700] Example prompt: What is the next action the delivery person should take if "It looks like it's going to be late" and "The roads are busy"? Generate a feedback statement.

[1701] In this way, delivery personnel can respond quickly and appropriately on site.

[1702] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1703] Step 1:

[1704] During delivery, the user inputs voice data into the smartphone, for example, "I'm going to be late, the roads are congested." This voice data is collected and sent from the smartphone to the server in real time.

[1705] Input: Delivery person's voice data

[1706] Output: Audio data sent to the server

[1707] Step 2:

[1708] The server converts the received voice data into text data using an automatic speech recognition engine (Google Cloud Speech-to-Text). Through this process, the voice data is converted into text information.

[1709] Input: Audio data

[1710] Output: Text data

[1711] Step 3:

[1712] The server analyzes the obtained text data using a natural language processing engine (spaCy or NLTK). Keywords in the text data (e.g., "I'm going to be late" or "The roads are congested") are extracted, and the topic and content of the utterance are identified.

[1713] Input: Text data

[1714] Output: Parsed topic and content information

[1715] Step 4:

[1716] The server uses emotion recognition AI (Microsoft Azure Emotion API) to recognize the emotions of the delivery person from the analysis results. Emotion recognition determines whether the delivery person is feeling anxious or stressed from the text data.

[1717] Input: Parsed topic and content information

[1718] Output: Recognized emotion data

[1719] Step 5:

[1720] The server monitors the delivery progress based on the analysis results and sentiment data, and if an abnormal progress (e.g., risk of delay) is detected, it proceeds to the next step.

[1721] Input: Recognized emotion data

[1722] Output: Progress determination result

[1723] Step 6:

[1724] The server uses a generative AI model (OpenAI GPT-4) to generate appropriate advice and suggestions. The prompt sentence is set as "What is the next action the delivery person should take if the delivery person is likely to be late and the roads are congested?" and the generated suggestions are created.

[1725] Input: Progress status determination result

[1726] Output: Generated advice and suggestions

[1727] Step 7:

[1728] The server sends the generated advice and suggestions to the terminal.

[1729] Input: Generated advice and suggestions

[1730] Output: Tips and suggestions sent to the terminal

[1731] Step 8:

[1732] The device displays the received advice and suggestions in real time. The delivery person can then refer to these advice and suggestions and select the appropriate action to carry out their task. Specifically, they can check for detours according to the example sentences displayed, or report delays to the customer.

[1733] Input: Tips and suggestions sent to your device

[1734] Output: Tips and suggestions displayed on the terminal

[1735] This allows delivery personnel to respond quickly and appropriately on site.

[1736] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1737] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1738] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1739] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1740] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1741] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1742] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1743] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1744] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1745] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1746] The system according to the present...

Claims

1. means for collecting and transmitting voice data to a server in real time; A means for converting the voice data received by the server into text data using an automatic voice recognition engine and analyzing the text data using a natural language processing engine; a means for monitoring the progress of the conference based on the analysis result by the server and detecting an abnormal progress; A means for the server to generate advice or suggestions based on the detection results and transmit the suggestions to the terminal; a means for displaying in real time the advice or suggestions received by the device; A means for the server to analyze past conference data, generate a conference playbook using artificial intelligence, and transmit the playbook to the terminal; The system includes means for displaying the conference playbook received by the terminal.

2. The system according to claim 1, wherein the system monitors topics where progress is stalled and the content of participants' comments based on the analyzed text data.

3. 2. The system according to claim 1, wherein the server displays materials and sample sentences for supporting the progress of a meeting based on a meeting playbook generated by the server.

Images