System

Abstract

A system is provided.SOLUTION: An assistant system for improving productivity of a meeting, comprising: means for acquiring a goal of the meeting from a user; means for analyzing voice data in real time and converting the voice data into text data; means for detecting a deviation from the meeting based on the text data and notifying the user of an alert; means for generating and presenting a related idea when a discussion stagnates; means for monitoring a progress time of the meeting and prompting closing; means for organizing action items extracted during the meeting and generating a to-do list; and means for analyzing previous meeting data and suggesting a next meeting agenda.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, many companies and organizations hold frequent meetings, but in many cases, these meetings are unproductive and it is difficult to reach concrete conclusions or action items. Major problems include discussions going off track, ideas stalling, and poor meeting time management. It is also difficult to efficiently compile agendas and to-do lists for the next meeting after a meeting. There is a need for a way to solve these issues and improve meeting productivity. [Means for solving the problem]

[0005] The present invention provides an assistant system for improving meeting productivity. This system includes a means for acquiring meeting goals from a user, a means for analyzing voice data in real time and converting it into text data, a means for detecting derailments in the meeting based on the text data and notifying the user of an alert, a means for generating and presenting related ideas when the discussion stalls, a means for monitoring the meeting progress and encouraging closing, a means for organizing action items extracted during the meeting and generating a to-do list, and a means for analyzing previous meeting data and proposing the next meeting agenda. This allows for efficient management of meeting progress and enables concrete results to be achieved.

[0006] A "meeting" is a gathering of multiple participants to discuss and make decisions about a specific topic.

[0007] "Productivity" is a concept that indicates the efficiency of results obtained using a certain amount of resources or time.

[0008] An "assistance system" is a combination of software and hardware designed to assist with a specific task or process.

[0009] "User" refers to an individual or organization that uses the system or service.

[0010] A "goal" refers to a specific purpose or objective to be achieved in a meeting.

[0011] "Audio data" refers to data that is a digital recording of audio collected through a microphone.

[0012] "Real-time" refers to the timeframe in which the system collects data and processes and analyzes it immediately.

[0013] "Text data" refers to character information converted from audio data.

[0014] A "derailment" occurs when the meeting discussion strays from the established goal.

[0015] An "alert" is a warning or notification that is issued when a specific condition is met.

[0016] A "discussion stall" refers to a situation in which no new ideas or progress are being made during a meeting and the conversation is not progressing.

[0017] An "idea" refers to a specific proposal or thought that helps solve a problem or achieve a goal.

[0018] "Closing" refers to the summarization and conclusions that take place near the end of a meeting.

[0019] "Action items" refer to specific tasks or work items decided upon in a meeting.

[0020] A "to-do list" is a list of tasks or action items that need to be completed.

[0021] An "agenda" refers to a schedule or order of proceedings that outlines the progress and topics of a meeting.

[0022] A "proposal" refers to a new approach or solution to a particular issue or problem. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[0045] 1. Setting meeting goals

[0046] The server accepts the goals specified by the user from the terminal at the start of the meeting and stores this goal information, which becomes the basis for all subsequent analysis and suggestions.

[0047] Example: When a user types into a terminal, "The goal of today's meeting is to decide on a release schedule for a new product," the server records this information and uses it as the basis for the entire process.

[0048] 2. Real-time analysis of voice data

[0049] The device transmits the voice data collected during the meeting to the server in real time. The server converts the voice data into text data using speech recognition technology. At the same time, the server analyzes the text data and determines whether it meets the goal.

[0050] Example: If the audio data of a meeting collected by a device includes the phrase "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes whether the agenda is in line with the goal.

[0051] 3. Derailment detection and alerts

[0052] If the server determines that the discussion is deviating from the meeting goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the device, which then notifies the user of the alert visually or audibly.

[0053] Example: If the conversation veers off topic, such as "about weekend events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from the goal."

[0054] 4. Support for generating ideas

[0055] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[0056] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about the name 'EcoPremium'?"

[0057] 5. Automatic time management and closing

[0058] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[0059] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[0060] 6. Organizing and automatically generating ToDos

[0061] The server extracts important action items discussed during the meeting based on the data and text collected during the meeting, organizes them systematically, and generates a to-do list. This to-do list is then sent to the device and provided to the user.

[0062] Example: After the meeting ends, the server generates the following action items: 1. Plan a promotional campaign 2. Collect feedback on the product prototype 3. Review the marketing strategy, and displays them on the device.

[0063] 7. Proposal for the next agenda

[0064] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[0065] Example: Based on the previous to-do list, an agenda for the next meeting is automatically generated and displayed on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[0066] As described above, through the embodiment of the present invention, the progress of a meeting can be managed efficiently and specific results can be obtained.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[0070] Step 2:

[0071] The terminal transmits the entered goal information to the server, which stores and organizes this information for use in subsequent processes.

[0072] Step 3:

[0073] The device uses a microphone to collect audio data during the meeting, which is then streamed to a server in real time.

[0074] Step 4:

[0075] The server uses speech recognition technology to convert the received voice data into text data, which is then stored for analysis.

[0076] Step 5:

[0077] The server uses the analyzed text data to assess whether the discussion is aligned with the meeting's goals, and if the topic deviates from the goals, the server generates an alert containing that information.

[0078] Step 6:

[0079] The server sends the generated alert to the terminal, which notifies the user of the alert visually or audibly.

[0080] Step 7:

[0081] The server detects an impasse when a discussion has not progressed for a certain period of time, and when it detects an impasse, it consults a database or past data to generate related ideas.

[0082] Step 8:

[0083] The server sends the generated idea as a text message to the terminal, which notifies the user of the idea.

[0084] Step 9:

[0085] The device monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting.

[0086] Step 10:

[0087] The server generates a message informing the terminal that the termination is imminent and sends it to the terminal, which then notifies the user with a message prompting the user to close.

[0088] Step 11:

[0089] The server extracts important action items discussed during the meeting based on the data collected during the meeting, and organizes the extracted items into a to-do list.

[0090] Step 12:

[0091] The server sends the generated ToDo list to the terminal, which displays the list to the user.

[0092] Step 13:

[0093] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss at the next meeting.

[0094] Step 14:

[0095] The server generates the next agenda and sends it to the terminal, which then proposes the next meeting agenda to the user.

[0096] This is the specific process flow, which allows the progress of the meeting to be managed efficiently and makes it possible to achieve specific results.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] Current meeting management systems have problems in that they are unable to adequately manage meeting progress or improve discussion productivity. Conventional systems make it difficult for users to monitor the progress of discussions in real time or immediately detect when a topic has gone off track. Furthermore, when a discussion stalls, users are required to find a solution themselves, which takes time and effort. Furthermore, organizing action items after a meeting and setting the next agenda is time-consuming, hindering efficient work.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes means for acquiring meeting goals from a user, means for analyzing voice data in real time and converting it into text data, means for detecting deviations in the meeting based on the text data and notifying the user of an alert, means for generating and presenting relevant proposals when the discussion stagnates, means for monitoring the progress of the meeting and encouraging closing, means for organizing important matters extracted during the meeting and generating waiting items, and means for analyzing the previous meeting data and proposing the content of the next meeting. This makes it possible to improve meeting productivity, efficiently manage the progress, and maximize the results of the discussion.

[0102] "Meeting goals" refers to the purpose of the meeting or the objectives to be achieved that are set by the user at the start of the meeting.

[0103] "Real-time analysis" refers to the process of instantly converting voice data into text data and rapidly analyzing its content.

[0104] "Text data" refers to data obtained as a result of converting voice data into text information.

[0105] The "derailment" detection method refers to a function that determines whether the discussion is deviating from the set goal of the meeting and notifies the user.

[0106] An "alert" is a message that notifies the user of important information or a warning.

[0107] "When the discussion stagnates" refers to a situation where the discussion does not progress for a certain period of time during a meeting.

[0108] "Relevant proposals" refer to specific ideas and solutions generated to further the discussion.

[0109] "Means to encourage closing" refers to a function that notifies users when the meeting is about to end and encourages them to wrap up the meeting.

[0110] "Key issues" refer to the most important action items or decisions discussed during the meeting.

[0111] "Queue items" refers to an organized list of action items decided during the meeting.

[0112] "Next meeting content" refers to the topics or agenda items to be discussed at the next meeting based on the previous meeting data.

[0113] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[0114] The system consists of a server and a terminal, and users interact with the server through the terminal. The system utilizes voice recognition technology, natural language processing technology, and generative AI models.

[0115] Hardware and Software Configuration

[0116] The servers are high-performance servers and can utilize the following cloud platforms:

[0117] Amazon Web Services (AWS)

[0118] Google Cloud Platform (GCP)

[0119] The terminals are typical user devices such as PCs, tablets, smartphones, etc. These terminals can access the server via an internet connection.

[0120] The speech recognition software uses Google Speech-to-Text API and IBM Watson Speech to Text to rapidly convert voice data into text data, while Python's Pandas and Numpy are used for data analysis, and NLTK and spaCy are used for automated natural language processing.

[0121] Specific operation of the system

[0122] 1. Setting meeting goals

[0123] The user inputs the goal of the meeting using the terminal, and the terminal sends this information to the server, which stores it in a database.

[0124] Example: A user types, "The goal of today's meeting is to decide on a release schedule for a new product." The server records this.

[0125] 2. Real-time analysis of voice data

[0126] During a meeting, the device collects audio data and streams it to a server in real time. The server converts the audio data into text data and stores the analysis results.

[0127] Example: If a user says, "We are discussing the target customers for a new product," the device collects the speech and the server converts it into text for analysis.

[0128] 3. Derailment detection and alerts

[0129] The server uses the text data to determine whether the discussion is deviating from the goal of the meeting. If so, the server generates an alert and sends it to the device. The device then notifies the user.

[0130] Example: When the conversation shifts to a topic unrelated to the meeting goal, such as "About the weekend events," the server generates an alert such as "Warning: The conversation is straying from the goal" and notifies the device.

[0131] 4. Support for stalled discussions and idea generation

[0132] The server monitors the progress of the meeting and detects impasses when the discussion has stalled for a certain period of time. The server generates relevant proposals and sends them to the terminal, which then presents them to the user.

[0133] Example: When the discussion is "We can't decide on a candidate for a new product name," the server suggests "For example, one name is 'EcoPremium,'" and the terminal displays this.

[0134] 5. Automatic time management and closing

[0135] The terminal monitors the progress of the meeting in real time. When the end time approaches, the terminal sends a notification to the server. The server generates a message prompting the user to close the meeting and sends it to the terminal. The terminal then notifies the user.

[0136] Example: Five minutes before the end of a meeting, the device displays the message, "Meeting will end soon. Please summarize the key points."

[0137] 6. Organizing and automatically generating ToDos

[0138] The server analyzes the data and text collected during the meeting, extracts important action items, and generates a to-do list based on this information and sends it to the device, which is then provided to the user.

[0139] Example: After the meeting ends, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and notifies the device.

[0140] 7. Proposal for the next agenda

[0141] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss in the next meeting. The server generates the next agenda and sends it to the device, which provides it to the user.

[0142] Example: Based on the previous to-do list, the device displays the following on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[0143] Prompt Sentence Examples

[0144] "What is the goal of today's meeting? Example: To decide on a new product release schedule."

[0145] "Determine whether the content of the conversation is aligned with the goal. For example, we are discussing target customers for a new product."

[0146] "Please summarize the key points of the meeting. For example, planning a promotional campaign, gathering feedback on a product prototype, or reviewing marketing strategies."

[0147] As described above, through the embodiment of the present invention, the progress of a meeting can be efficiently managed and specific results can be obtained.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] Program processing flow

[0150] Step 1: Set meeting goals

[0151] 1. Input: The user inputs the goal of the meeting into the terminal.

[0152] 2. Operation: The device sends the entered goal information to the server, which stores this information in a database and uses it as the basis for subsequent analysis.

[0153] 3. Output: The server stores the goal information, which is the baseline for the entire meeting.

[0154] Specific behavior:

[0155] The user inputs, "The goal of today's meeting is to decide on a release schedule for the new product."

[0156] The terminal sends this information to the server, which stores it in a database.

[0157] Step 2: Real-time analysis of audio data

[0158] 1. Input: The audio data that users say during a meeting.

[0159] 2. Operation: The device collects audio data during the meeting and streams it to the server in real time. The server then uses speech recognition technology to convert the audio data into text data.

[0160] 3. Output: The server stores the converted text data and uses it as the basis for analysis.

[0161] Specific behavior:

[0162] The device collects audio such as, "We are discussing target customers for a new product."

[0163] The server converts this into text and stores the analysis results.

[0164] Step 3: Derailment detection and alerting

[0165] 1. Input: Text data acquired by the server in real time.

[0166] 2. Operation: The server analyzes the text data and determines whether the discussion is in line with the meeting goals. If it detects a deviation, it generates an alert and sends it to the device.

[0167] 3. Output: The device notifies the user of the alert visually or audibly.

[0168] Specific behavior:

[0169] If the conversation turns to a topic unrelated to the goal, such as "about weekend events," the server will determine that the conversation has "digressed."

[0170] The server generates an alert and notifies the device, "Warning: The conversation is straying from the goal."

[0171] Step 4: Resolving stalled discussions and supporting idea generation

[0172] 1. Input: Text data and discussion progress analyzed in real time by the server.

[0173] 2. Operation: The server monitors the progress of the discussion and generates a related proposal if it stalls for a certain period of time. The proposal is then sent to the terminal.

[0174] 3. Output: The device notifies the user of the suggestion.

[0175] Specific behavior:

[0176] When the discussion is "No candidate for a new product name can be decided," the server generates a candidate called "EcoPremium."

[0177] The terminal presents this to the user.

[0178] Step 5: Automatic time management and closing

[0179] 1. Input: The meeting progress time recorded by the device.

[0180] 2. Operation: The terminal monitors the progress of the meeting and notifies the server when the end time approaches. The server generates a closing message and sends it to the terminal.

[0181] 3. Output: The terminal notifies the user with a closing message.

[0182] Specific behavior:

[0183] Five minutes before the end of the meeting, the device will display the message, "The meeting will end soon. Please summarize the main points."

[0184] Step 6: Organize and auto-generate to-dos

[0185] 1. Input: Text data collected during the meeting, action item extraction results.

[0186] 2. Operation: The server analyzes the data, extracts important action items, and generates a to-do list, which is then sent to the device.

[0187] 3. Output: The device notifies the user of the ToDo list.

[0188] Specific behavior:

[0189] After the meeting, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and sends it to the device.

[0190] Step 7: Propose the next agenda

[0191] 1. Input: Previous meeting data and generated to-do list.

[0192] 2. Operation: The server analyzes these data and identifies the topics to be discussed in the next meeting. The generated agenda is sent to the device.

[0193] 3. Output: The terminal notifies the user of the next agenda item.

[0194] Specific behavior:

[0195] Based on the previous to-do list, an agenda for the next meeting is generated: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy, and is displayed on the terminal.

[0196] (Application example 1)

[0197] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0198] Existing assistant systems that improve meeting productivity are designed for use in general office environments, making it difficult to meet the unique needs of specific environments, such as factory production lines. Furthermore, there is a lack of means to efficiently implement functions such as progress management of work instructions, derailment detection, and related idea generation within factories. As a result, work efficiency within factories declines, making productivity improvement a challenge.

[0199] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0200] In this invention, the server includes: means for acquiring meeting goals from a user; means for analyzing voice data in real time and converting it into text data; means for detecting deviations in the meeting based on the text data and sending an alert to the user; means for generating and presenting related ideas when the discussion stalls; means for monitoring the progress of the meeting and encouraging closing; means for organizing action items extracted during the meeting and generating a to-do list; means for analyzing data from the previous meeting and proposing an agenda for the next meeting; means for collecting work instructions in the factory from a user and monitoring the progress of work based on the instructions; and means for generating an alert and notifying workers when work deviates from the plan. This makes it possible to improve work efficiency and productivity even in the factory.

[0201] An "assistance system that improves meeting productivity" is a support system that improves the progress and efficiency of discussions in meetings.

[0202] A "meeting goal" is a target that participants are trying to achieve, set at the start of the meeting.

[0203] "Audio data" refers to data recorded in digital format of conversations during a meeting.

[0204] "Text data" refers to character string information converted from voice data using voice recognition technology.

[0205] "Detecting digressions" means identifying when a discussion deviates from its established goals.

[0206] "Notifying an alert" means notifying the user of a warning by visual or audio means.

[0207] "Generating related ideas" means that if the discussion does not progress, the server automatically generates new proposals and solutions.

[0208] "Monitoring progress time" means recording and managing the time that has elapsed since the start of a meeting or task.

[0209] "Encouraging closure" refers to informing participants that a meeting or task is coming to an end and encouraging them to wrap things up.

[0210] "Action items" refer to important items that were discussed during the meeting or specific next steps to take.

[0211] A "ToDo list" is a list of tasks that need to be done.

[0212] "Factory work instructions" refers to instructions and orders for specific work to be done within a factory.

[0213] "Monitoring work progress" refers to monitoring whether work being done in a factory is progressing as planned.

[0214] "Notifying workers" refers to informing on-site staff involved in the work of important information or warnings.

[0215] The present invention is an assistant system for improving work efficiency in a factory, and a specific embodiment thereof is described below. The system is realized by linking a server and a terminal.

[0216] 1. Setting goals for work instructions

[0217] The server accepts the user's specified goal from the terminal at the start of the task and stores this goal information, which becomes the basis for all subsequent analysis and proposals.

[0218] Example: When a user types into a terminal, "Today's work goal is to check the quality of the new product," the server records this information and uses it as the basis for the entire process.

[0219] 2. Real-time analysis of voice data

[0220] The device transmits the voice data collected during work to the server in real time, and the server converts the voice data into text data using voice recognition technology. At the same time, the server analyzes the text data and determines whether it matches the goal.

[0221] Example: If the audio data collected by the device about a task is, "We are conducting a visual inspection of a new product," the server immediately converts this into text data and analyzes whether the task is in line with the goal.

[0222] 3. Derailment detection and alerts

[0223] If the server determines that the task is deviating from the set goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[0224] Example: If the conversation turns to "About this weekend's events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from goal."

[0225] 4. Support for generating ideas

[0226] The server monitors when the work has not progressed for a certain period of time, detects impasses, and generates related ideas as needed, sending them to the terminal and suggesting them to the user.

[0227] Example: If the problem is "we can't decide on a product name candidate", the server will display a suggestion on the terminal, such as "How about the name 'Quality Plus'?"

[0228] 5. Automatic time management and closing

[0229] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[0230] Example: Five minutes before the scheduled end of a task, the device displays the message, "Your task will be completed soon. Please summarize the main points."

[0231] 6. Organizing and automatically generating ToDos

[0232] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[0233] Example: After the task is completed, the server generates a list such as "Next action items: 1. Collect feedback on the product prototype 2. Propose quality improvements" and displays it on the terminal.

[0234] 7. Proposal for the next agenda

[0235] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, and sends it to the terminal to suggest to the user.

[0236] Example: Based on the previous ToDo list, an agenda item titled "Next work agenda: 1. Evaluate feedback 2. Implement quality improvement measures" is automatically generated and displayed on the device.

[0237] This system uses Python to control the entire program and the "speech_recognition" voice recognition library. The microphones used to collect the voice data are highly sensitive, suitable for use in factories. A specific use case is the issuance of work instructions for product quality checks.

[0238] Example prompt sentence:

[0239] The next work order is:

[0240] 1. Check the quality of all items of new product A

[0241] 2. Update checklist items

[0242] 3. Creation of a report on the check results

[0243] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0244] Step 1:

[0245] The user inputs the task goal into the terminal. The terminal sends this information to the server. The server saves the received goal information. This saved information becomes the basis for all subsequent analysis and suggestions.

[0246] Input: Task goal information entered by the user

[0247] Data processing: The server saves the goal information

[0248] Output: Saved goal information

[0249] Step 2:

[0250] The device collects voice data while working and transmits it in real time to a server, which uses voice recognition technology to convert the voice data into text data, which the server then analyzes to determine whether it meets the goal.

[0251] Input: Audio data collected during work

[0252] Data processing: The server converts the voice data into text data and analyzes the text data.

[0253] Output: Analyzed text data and goal conformance assessment results

[0254] Step 3:

[0255] The server generates a derailment alert based on the converted text data if the task deviates from the goal and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[0256] Input: Text data and goal conformance assessment results

[0257] Data processing: The server generates derailment alerts

[0258] Output: Derailment alert sent to the terminal

[0259] Step 4:

[0260] The server monitors when the work has not progressed for a certain period of time and detects impasses. If necessary, the server generates related ideas, sends them to the terminal, and suggests them to the user.

[0261] Input: Text data and time course information

[0262] Data processing: The server detects impasses and generates ideas

[0263] Output: Related ideas sent to your device

[0264] Step 5:

[0265] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[0266] Input: Start time and expected end time of work

[0267] Data processing: The server generates a closing message

[0268] Output: Closing message sent to the terminal

[0269] Step 6:

[0270] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[0271] Input: Data or text captured during work

[0272] Data processing: The server extracts action items and generates a to-do list

[0273] Output: To-do list sent to the device

[0274] Step 7:

[0275] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, sends it to the terminal, and proposes it to the user.

[0276] Input: Previous work data and generated ToDo list

[0277] Data processing: The server generates the next agenda

[0278] Output: Next agenda sent to terminal

[0279] The above is the specific processing flow of the program of the system that realizes the application example.

[0280] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0281] The present invention relates to an assistant system that improves the productivity of meetings, and furthermore, it combines an emotion engine to recognize and analyze the user's emotional state, thereby more effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[0282] 1. Setting meeting goals

[0283] The server accepts the goals specified by the user from the terminal at the start of the meeting, stores this goal information, and uses this information as the basis for all subsequent analysis and suggestions.

[0284] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as the basis for the entire process.

[0285] 2. Emotion recognition by emotion engine

[0286] The device uses sensors to collect the user's facial expressions and tone of voice during the meeting. This data is sent in real time to the server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[0287] Example: If the user is recognized as feeling stressed, the emotion engine analyzes the information and sends it to the server.

[0288] 3. Real-time analysis of audio data

[0289] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the data into text data for analysis.

[0290] Example: If the meeting audio data collected by the device includes the sentence, "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes it.

[0291] 4. Derailment detection and alerts

[0292] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data, and if the topic deviates from the goal, it generates a derailment alert and sends it to the terminal. The terminal notifies the user of this alert.

[0293] Example: If the conversation veers off topic, such as "about weekend events," the device will display a notification saying, "Warning: Conversation is drifting away from goal."

[0294] 5. Support for generating ideas

[0295] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[0296] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about 'EcoPremium'?"

[0297] 6. Automatic time management and closing

[0298] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[0299] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[0300] 7. Organizing and automatically generating ToDos

[0301] The server extracts important action items discussed during the meeting based on the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[0302] Example: After the meeting ends, the server generates "Next Tasks: 1. Promotion campaign plan 2. Collect feedback on product prototype 3. Review marketing strategy" and displays it on the terminal.

[0303] 8. Proposal for the next agenda

[0304] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[0305] Example: Based on the previous ToDo list, the following agenda is automatically generated and displayed on the device: "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[0306] 9. Moderating discussions based on emotional states

[0307] The server adjusts the progress of the discussion based on the user's emotional state and provides relaxation and motivational suggestions as needed. For example, if the server detects that the user is feeling stressed, it will provide relaxation suggestions.

[0308] Example: If a user is feeling stressed, the server displays a suggestion on the device such as "Take a deep breath and refresh yourself."

[0309] As described above, through the embodiment of the present invention, the progress of a meeting is efficiently managed and appropriate feedback based on the emotion engine is provided, thereby making it possible to achieve concrete results.

[0310] The processing flow will be explained below.

[0311] Step 1:

[0312] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[0313] Step 2:

[0314] The terminal sends the entered goal information to the server, which stores the goal information and organizes it for use in subsequent processes, so that the progress of the entire meeting can be adjusted based on the goal.

[0315] Step 3:

[0316] During meetings, the device uses a microphone and camera to collect voice and facial expression data, which is then streamed to a server in real time.

[0317] Step 4:

[0318] The server converts the received voice data into text data using voice recognition technology, while simultaneously analyzing facial expression data and voice tone to recognize the user's emotional state.

[0319] Step 5:

[0320] The server evaluates whether the discussion is in line with the meeting's goals based on the analyzed text data. If the topic strays from the goal, it generates a derailment alert. It also takes into account the user's emotional state, and generates a message suggesting relaxation if, for example, stress levels are high.

[0321] Step 6:

[0322] The server sends the generated alert and suggestion messages to the terminal, which notifies the user of these messages visually or audibly.

[0323] Example: If the discussion veers off course to "weekend events" and the user is feeling stressed, the device will display "Warning: The conversation is drifting away from the goal. Take a deep breath and refresh yourself."

[0324] Step 7:

[0325] The server monitors when the discussion has not progressed for a certain period of time and detects an impasse. When it detects an impasse, it generates related ideas and sends them to the terminal.

[0326] Step 8:

[0327] The terminal presents the generated ideas to the user, helping the user to restart the discussion based on the new ideas.

[0328] Example: When a new product name cannot be decided, the server generates a suggestion such as "How about 'EcoPremium'?" and the terminal displays this to the user.

[0329] Step 9:

[0330] The terminal monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting. The server then generates a message informing the terminal that the meeting is about to end and sends it to the terminal.

[0331] Step 10:

[0332] The terminal notifies the user of a message prompting the user to close the meeting, and the user begins work to summarize the main points of the meeting.

[0333] Example: Five minutes before the end, the device displays, "We're almost done. Let's summarize the main points."

[0334] Step 11:

[0335] The server extracts important action items discussed during the meeting from the text data acquired during the meeting, organizes them into a to-do list, and sends it to the device.

[0336] Step 12:

[0337] The device displays the generated to-do list to the user, helping them organize their work after the meeting.

[0338] Example: After the meeting, the following tasks are displayed: 1. Plan a promotional campaign 2. Collect feedback on product prototypes 3. Review marketing strategy.

[0339] Step 13:

[0340] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal.

[0341] Step 14:

[0342] The terminal will suggest the next meeting agenda to the user, allowing the user to effectively prepare for the next discussion.

[0343] Example: The next agenda is proposed as follows: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion.

[0344] These are the specific processing steps of the meeting assistant system that combines the emotion engine. This system not only efficiently and effectively manages the progress of meetings, but also provides appropriate feedback based on the user's emotional state, enabling concrete results to be achieved.

[0345] Example 2

[0346] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0347] Conventional meeting management systems have the problem of being unable to grasp the progress of the meeting or the user's emotional state in real time and provide effective support based on that information. This poses the problem of meetings easily going off track or discussions stalling, reducing the overall productivity of the meeting. Furthermore, because appropriate suggestions and adjustments are not made based on the user's emotional state, there is a risk that users' stress will increase and their concentration and motivation will decrease.

[0348] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0349] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, and means for recognizing the user's emotional state in real time and adjusting the progress of the discussion based on the emotion. This makes it possible to prevent derailment and stagnation during meetings, support effective idea generation and action item organization, and provide appropriate feedback according to the user's emotional state.

[0350] A "meeting goal" is a specific objective or purpose that you are trying to achieve through the meeting.

[0351] "User" refers to a person who operates this system and participates in meetings.

[0352] "Audio data" refers to digital data of audio collected during a meeting.

[0353] "Text data" is digital data that has been analyzed and converted into text information from audio data.

[0354] A "digression" refers to a topic that strays from the goal of the meeting or a discussion that goes off-target.

[0355] An "alert" is a warning message that notifies the user of derailments or problems detected by the system.

[0356] "Ideas" refer to new proposals or solutions that advance the discussion.

[0357] "Closing" is the process of concluding a meeting and summarizing outcomes and action items.

[0358] "Action items" are items that are discussed during a meeting and turned into concrete actions or plans.

[0359] A "ToDo list" is an organized list of action items decided during a meeting.

[0360] An "agenda" refers to the specific topics or items to be discussed at the next meeting.

[0361] "Emotional state" refers to the user's current psychological state or mood.

[0362] The "means for adjusting the progress of the discussion" is a function that appropriately adjusts the progress and content of the meeting based on the user's emotional state.

[0363] The present invention relates to an assistant system that improves meeting productivity, and provides technical means for recognizing and analyzing the user's emotional state and effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[0364] 1. Setting meeting goals

[0365] The server accepts the meeting goals specified by the user from the terminal at the start of the meeting and stores this goal information, which is used as the basis for all subsequent analysis and suggestions.

[0366] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as a reference for the entire process. This is done so that the server can store the goal information in a database.

[0367] 2. Emotion recognition by emotion engine

[0368] During a meeting, the device captures the user's facial expressions with a camera and collects the user's tone of voice with a microphone. This data is then sent in real time to a server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[0369] Example: The device's camera and microphone are activated to collect data on the user's facial expressions and voice. The server's emotion engine recognizes that the user is feeling stressed and stores this information in a database.

[0370] 3. Real-time analysis of audio data

[0371] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text data for analysis.

[0372] Example: The device's microphone is activated and captures meeting audio such as "We are discussing target customers for a new product." The server converts this into text data and stores it in a database.

[0373] 4. Derailment detection and alerts

[0374] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data. If the topic deviates from the goal, it generates a digression alert and sends it to the device. The device notifies the user of this alert.

[0375] Example: The server analyzes text data and detects deviations such as "About weekend events." The device displays a notification such as "Warning: The conversation is deviating from the goal."

[0376] 5. Support for generating ideas

[0377] The server monitors when the discussion has not progressed for a certain period of time, detects the impasse, generates related ideas, and sends them to the terminal to suggest to the user.

[0378] Example: The server detects when a meeting is stalling and displays suggestions on the terminal, such as "How about 'EcoPremium'?"

[0379] 6. Automatic time management and closing

[0380] The terminal monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, for example, five minutes before the end, and sends it to the terminal. The terminal then notifies the user of this message.

[0381] Example: The device monitors the progress time and displays the message "The meeting will end soon. Please summarize the main points" five minutes before the scheduled end time.

[0382] 7. Organizing and automatically generating ToDos

[0383] The server extracts important action items from the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[0384] Example: The server generates a list from the discussion content, such as "Next To Do: 1. Plan a promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy," and displays it on the device.

[0385] 8. Proposal for the next agenda

[0386] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[0387] Example: The server analyzes the previous data and generates a list such as "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion" and displays it on the terminal.

[0388] 9. Moderating discussions based on emotional states

[0389] The server adjusts the progress of the discussion based on the user's emotional state, providing relaxation and motivational suggestions as needed. For example, if it detects that the user is feeling stressed, it will provide relaxation suggestions.

[0390] Example: Your device might display a suggestion such as "Take a deep breath and refresh yourself."

[0391] As a result, this assistant system achieves the technical features described in (Claim 1) to (Claim 3), efficiently manages the progress of meetings, and provides appropriate feedback based on the user's emotional state, thereby achieving concrete results.

[0392] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0393] Step 1: Set meeting goals

[0394] Input: The user enters the meeting goal into the device.

[0395] Specific behavior: The user enters "Today's goal is to determine the release schedule for the new product" in the text box and clicks the submit button.

[0396] Data processing: The device receives the entered goal information and sends it to the server.

[0397] Output: Goal information is saved on the server.

[0398] Specific operation: The server records the received goal information in the database and uses it as the basis for subsequent processing.

[0399] Step 2: Recognizing user emotions with the emotion engine

[0400] Input: The user's facial expressions and tone of voice are collected via a camera and microphone.

[0401] What it does: The device's camera captures the user's facial expressions, and the microphone collects the tone of their voice.

[0402] Data processing: The device transmits the collected facial expression and voice data to the server in real time, and the server uses an emotion engine to analyze the data and recognize the user's emotional state.

[0403] Output: The user's emotional state is stored on the server.

[0404] Specific operation: The server's emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[0405] Step 3: Real-time analysis of audio data

[0406] Input: The microphone collects audio data during the meeting.

[0407] Specific operation: The device's microphone is activated and recording the speech of meeting participants.

[0408] Data processing: The device sends the collected voice data to a server in real time. The server uses voice recognition technology to convert the voice data into text data for analysis.

[0409] Output: Text data is generated on the server.

[0410] Specific operation: The server generates text data such as "We are discussing target customers for a new product."

[0411] Step 4: Derailment detection and alerting

[0412] Input: The text data described above.

[0413] Specific behavior: The server analyzes the text data and evaluates whether the discussion is in line with the meeting goals.

[0414] Data processing: The server detects derailments based on text data and generates alerts.

[0415] Output: A derailment alert is sent from the server to the device.

[0416] Specific behavior: The device notifies the user with an alert such as "Warning: The conversation is straying from the goal."

[0417] Step 5: Ideation support

[0418] Input: Meeting progress and stall information.

[0419] Specific operation: The server monitors the progress of the meeting and detects when no progress has been made for a certain period of time.

[0420] Data processing: The server generates relevant ideas and sends them to the device.

[0421] Output: The suggestions are displayed in the terminal.

[0422] What it does: Your device will display suggestions like, "How about 'EcoPremium'?"

[0423] Step 6: Automatic time management and closing

[0424] Input: Meeting start time.

[0425] Specific behavior: The device monitors the meeting progress time.

[0426] Data processing: The terminal notifies the server that the scheduled end time is approaching. The server generates a closing message and sends it to the terminal.

[0427] Output: A closing message is printed to the terminal.

[0428] What happens: The device displays the message "Meeting will end soon. Please summarize the key points."

[0429] Step 7: Organizing and automatically generating to-dos

[0430] Input: Action item information captured during the meeting.

[0431] What it does: The server extracts important action items from the meeting discussion.

[0432] Data processing: The server organizes the action items and generates a to-do list.

[0433] Output: The to-do list is sent to the device.

[0434] Specific actions: The device will display "Next To Do: 1. Plan promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy."

[0435] Step 8: Propose the next agenda

[0436] Input: Previous meeting data and generated to-do list.

[0437] What happens: The server analyzes the previous data and identifies the next topic to discuss.

[0438] Data processing: The server generates the next agenda and sends it to the device.

[0439] Output: The agenda is displayed on the terminal.

[0440] Specific operation: The terminal displays "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[0441] Step 9: Adjust the discussion based on your emotional state

[0442] Input: User emotion data.

[0443] Specific behavior: The server continuously monitors the user's emotional state.

[0444] Data processing: The server generates appropriate suggestions based on the emotional state and sends them to the device.

[0445] Output: The sentiment-based suggestions will be displayed on your device.

[0446] What it does: Your device will display a suggestion such as "Take a deep breath and refresh yourself."

[0447] The above is the flow of processing steps for how the program in this system processes and calculates data to generate specific output. At each step, the server and terminals work together to perform specific operations to improve the user's meeting experience.

[0448] (Application example 2)

[0449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0450] Efficient and productive meetings are essential in modern manufacturing. However, productivity is often hindered by digressions, stalled discussions, and the emotional state of workers during meetings. This can lead to a decline in the overall efficiency of manufacturing operations and failure to achieve production targets. In particular, a lack of appropriate responses due to changes in emotional states is often the cause. Therefore, there is a need for a system that supports efficient meeting conduct and provides optimal feedback by taking into account the emotional state of workers.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0452] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, means for recognizing the user's emotional state using an emotion engine and adjusting the discussion in real time, and means for proposing relaxation and motivation improvement based on the emotional state. This prevents derailment and stagnation in the discussion during the meeting and enables optimal responses based on the emotional state of the participants.

[0453] The "means for acquiring the goal of the meeting from the user" is a function for inputting and saving the goal set by the user at the start of the meeting into the server.

[0454] "Means for analyzing voice data in real time and converting it into text data" refers to a technology that converts voice data collected by a microphone during a meeting into text data in real time.

[0455] "Means of detecting derailment in meetings based on text data and notifying users of an alert" is a function that issues a warning to users if the discussion deviates from the goal based on analyzed text data.

[0456] "Means for generating and presenting related ideas when discussions stagnate" is a function that detects situations where discussions are not progressing and provides users with related suggestions and ideas.

[0457] "Means to monitor the progress of the meeting and encourage closing" is a function that monitors the progress of the meeting and encourages the summary of the main points near the end.

[0458] "A means to organize action items extracted during meetings and generate a to-do list" is a function that extracts important tasks discussed during meetings and generates a structured to-do list.

[0459] "A means of analyzing the data from the previous meeting and proposing the agenda for the next meeting" is a function that analyzes the data and to-do list from the previous meeting and automatically proposes the next meeting topic.

[0460] "Means of using an emotion engine to recognize the user's emotional state and adjust the discussion in real time" is a technology that uses data collected through sensors to analyze the user's emotions and instantly adjust the progress of the discussion.

[0461] The "means for suggesting relaxation techniques and motivation improvement based on the emotional state" is a function that suggests relaxation techniques and motivation improvement as needed based on the user's emotional state.

[0462] The present invention is a system for managing the progress of a meeting using a factory robot and providing feedback based on the emotional state of a worker. The system includes the following means.

[0463] 1. Setting meeting goals

[0464] The server receives the goal specified by the user from the terminal at the start of the meeting. This information is stored on the server and used to progress the entire meeting.

[0465] Example: If you set "Today's production target is to manufacture 1,000 units of product A," the server will record this and use it as the basis for the entire process.

[0466] 2. Real-time analysis of voice data

[0467] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text data for analysis.

[0468] Hardware / software used: Microphone, Python, speech_recognition

[0469] Example: When you say, "There's a delay in parts supply. What should we do?" the server converts this into text in real time and records it.

[0470] 3. Derailment detection and alerts

[0471] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data, and if the topic deviates from the goal, it generates a derailment alert and sends it to the terminal. The terminal notifies the user of this alert.

[0472] Example: If the conversation strays to something like "weekend events," the device will notify you, "We're straying from the current topic."

[0473] 4. Support for generating ideas

[0474] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[0475] Example: If no improvements can be found for a production line, the server might suggest, "For example, how about increasing automation?"

[0476] 5. Automatic time management and closing

[0477] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[0478] Example: Five minutes before the end, announce, "The meeting will end soon. Please summarize the main points."

[0479] 6. Organizing and automatically generating ToDos

[0480] The server extracts important action items discussed during the meeting based on the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[0481] Example: After the meeting ends, the server generates "Next ToDo: 1. Review parts supply 2. Adjust production line" and displays it on the terminal.

[0482] 7. Proposal for the next agenda

[0483] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[0484] Example: Automatically generate and display "Next agenda: 1. Measures to improve parts supply 2. Trial operation of new production line."

[0485] 8. Emotional state recognition and discussion coordination using an emotion engine

[0486] The device uses sensors to collect the user's facial expressions and tone of voice during the meeting and transmits this data in real time to the server, which then uses an emotion engine to analyze this data and recognize the user's current emotional state.

[0487] Hardware / software used: Webcam, Python, dlib, FER

[0488] Example: If a worker is feeling tired, the emotion engine will recognize this and the server will suggest, "Take a deep breath."

[0489] Example prompt sentence:

[0490] How will you deal with the delay in parts supply?

[0491] (The robot assistant converts this into text and uses emotion recognition to detect that the worker is feeling stressed.)

[0492] Take a deep breath, refresh yourself, and then think of ideas for improving parts supply.

[0493] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0494] Step 1:

[0495] Meeting goal setting

[0496] Input: The meeting goal specified by the user from their device.

[0497] Data processing: The server receives and stores the goal information entered by the user.

[0498] Output: Goal information saved on the server.

[0499] Specific operation: The user enters "Today's production target is to manufacture 1,000 units of product A" into the terminal and presses the send button. The terminal sends this information to the server, which records it.

[0500] Step 2:

[0501] Audio data collection and real-time analysis

[0502] Input: Audio data during the meeting.

[0503] Data processing: The device collects voice data through the microphone and transmits it to the server in real time. The server then converts the data into text using voice recognition technology.

[0504] Output: Meeting audio converted to text.

[0505] How it works: The device uses a microphone to capture audio during a meeting and sends it to the server, which then uses Python and the speech_recognition library to analyze the audio and turn it into text data.

[0506] Step 3:

[0507] Derailment detection and alert notification

[0508] Input: Parsed text data and user-defined goals.

[0509] Data processing: The server evaluates whether the discussion is in line with the meeting goals based on the text data. If the topic deviates from the goals, an alert is generated and sent to the device.

[0510] Output: Alert notification for discussions that deviate from the goal.

[0511] Specific behavior: The server analyzes the text data, and if the discussion strays from the topic of "weekend events," it sends an alert to the device saying "We're straying from the current topic," and the device displays this notification to the user.

[0512] Step 4:

[0513] Support for generating ideas

[0514] Input: A stalled discussion situation.

[0515] Data processing: The server monitors if the discussion has not progressed for a certain period of time and generates related ideas as needed.

[0516] Output: Related ideas presented to the user.

[0517] How it works: If the discussion stalls, the server uses a generative AI model to generate ideas and sends them to the device, which then suggests to the user, "For example, how about increasing automation?"

[0518] Step 5:

[0519] Automatic time management and closing notifications

[0520] Input: Meeting duration.

[0521] Data processing: The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server then generates a message to prompt the device to close the meeting and sends it to the device.

[0522] Output: Notice of closing.

[0523] Specific operation: The device monitors the meeting time and notifies the server 5 minutes before the end. The server sends a message to the device saying "The meeting will end soon. Please summarize the main points." The device then displays this message to the user.

[0524] Step 6:

[0525] Organize and auto-generate to-dos

[0526] Input: Action items discussed during the meeting.

[0527] Data processing: The server extracts important action items from the data acquired during the meeting and generates a to-do list.

[0528] Output: The generated to-do list.

[0529] Specific operation: After the meeting ends, the server generates a list of "Next To-Dos: 1. Review parts supply 2. Adjust production line" and sends it to the terminal. The terminal displays it to the user.

[0530] Step 7:

[0531] Proposal for next agenda

[0532] Input: Previous meeting data and generated to-do list.

[0533] Data processing: The server analyzes the previous meeting data and identifies the topics to be discussed in the next meeting. It generates the next agenda and sends it to the device.

[0534] Output: The proposed next agenda.

[0535] Specific operation: The server analyzes the previous data, generates "Next agenda: 1. Measures to improve parts supply 2. Trial operation of new production line" and sends it to the terminal. The terminal displays it to the user.

[0536] Step 8:

[0537] Emotional state recognition and discussion coordination using an emotion engine

[0538] Input: The user's facial expressions and tone of voice.

[0539] Data processing: The device transmits data collected by sensors to the server in real time, and the server analyzes the data using an emotion engine to recognize the user's emotional state.

[0540] Output: Suggestions and alerts based on the user's emotional state.

[0541] How it works: The device uses a webcam and microphone to capture facial expressions and tone of voice, and sends them to the server. The server then analyzes them using Python, dlib, and FER, and suggests "take a deep breath" if the user is feeling stressed.

[0542] Example prompt sentence:

[0543] How will you deal with the delay in parts supply?

[0544] (The robot assistant converts this into text and uses emotion recognition to detect that the worker is feeling stressed.)

[0545] Take a deep breath, refresh yourself, and then think of ideas for improving parts supply.

[0546] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0547] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0548] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0549] [Second embodiment]

[0550] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0551] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0552] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0553] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0554] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0555] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0556] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0557] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0558] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0559] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0560] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0561] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0562] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[0563] 1. Setting meeting goals

[0564] The server accepts the goals specified by the user from the terminal at the start of the meeting and stores this goal information, which becomes the basis for all subsequent analysis and suggestions.

[0565] Example: When a user types into a terminal, "The goal of today's meeting is to decide on a release schedule for a new product," the server records this information and uses it as the basis for the entire process.

[0566] 2. Real-time analysis of voice data

[0567] The device transmits the voice data collected during the meeting to the server in real time. The server converts the voice data into text data using speech recognition technology. At the same time, the server analyzes the text data and determines whether it meets the goal.

[0568] Example: If the audio data of a meeting collected by a device includes the phrase "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes whether the agenda is in line with the goal.

[0569] 3. Derailment detection and alerts

[0570] If the server determines that the discussion is deviating from the meeting goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the device, which then notifies the user of the alert visually or audibly.

[0571] Example: If the conversation veers off topic, such as "about weekend events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from the goal."

[0572] 4. Support for generating ideas

[0573] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[0574] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about the name 'EcoPremium'?"

[0575] 5. Automatic time management and closing

[0576] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[0577] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[0578] 6. Organizing and automatically generating ToDos

[0579] The server extracts important action items discussed during the meeting based on the data and text collected during the meeting, organizes them systematically, and generates a to-do list. This to-do list is then sent to the device and provided to the user.

[0580] Example: After the meeting ends, the server generates the following action items: 1. Plan a promotional campaign 2. Collect feedback on the product prototype 3. Review the marketing strategy, and displays them on the device.

[0581] 7. Proposal for the next agenda

[0582] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[0583] Example: Based on the previous to-do list, an agenda for the next meeting is automatically generated and displayed on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[0584] As described above, through the embodiment of the present invention, the progress of a meeting can be managed efficiently and specific results can be obtained.

[0585] The processing flow will be explained below.

[0586] Step 1:

[0587] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[0588] Step 2:

[0589] The terminal transmits the entered goal information to the server, which stores and organizes this information for use in subsequent processes.

[0590] Step 3:

[0591] The device uses a microphone to collect audio data during the meeting, which is then streamed to a server in real time.

[0592] Step 4:

[0593] The server uses speech recognition technology to convert the received voice data into text data, which is then stored for analysis.

[0594] Step 5:

[0595] The server uses the analyzed text data to assess whether the discussion is aligned with the meeting's goals, and if the topic deviates from the goals, the server generates an alert containing that information.

[0596] Step 6:

[0597] The server sends the generated alert to the terminal, which notifies the user of the alert visually or audibly.

[0598] Step 7:

[0599] The server detects an impasse when a discussion has not progressed for a certain period of time, and when it detects an impasse, it consults a database or past data to generate related ideas.

[0600] Step 8:

[0601] The server sends the generated idea as a text message to the terminal, which notifies the user of the idea.

[0602] Step 9:

[0603] The device monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting.

[0604] Step 10:

[0605] The server generates a message informing the terminal that the termination is imminent and sends it to the terminal, which then notifies the user with a message prompting the user to close.

[0606] Step 11:

[0607] The server extracts important action items discussed during the meeting based on the data collected during the meeting, and organizes the extracted items into a to-do list.

[0608] Step 12:

[0609] The server sends the generated ToDo list to the terminal, which displays the list to the user.

[0610] Step 13:

[0611] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss at the next meeting.

[0612] Step 14:

[0613] The server generates the next agenda and sends it to the terminal, which then proposes the next meeting agenda to the user.

[0614] This is the specific process flow, which allows the progress of the meeting to be managed efficiently and makes it possible to achieve specific results.

[0615] Example 1

[0616] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] Current meeting management systems have problems in that they are unable to adequately manage meeting progress or improve discussion productivity. Conventional systems make it difficult for users to monitor the progress of discussions in real time or immediately detect when a topic has gone off track. Furthermore, when a discussion stalls, users are required to find a solution themselves, which takes time and effort. Furthermore, organizing action items after a meeting and setting the next agenda is time-consuming, hindering efficient work.

[0618] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0619] In this invention, the server includes means for acquiring meeting goals from a user, means for analyzing voice data in real time and converting it into text data, means for detecting deviations in the meeting based on the text data and notifying the user of an alert, means for generating and presenting relevant proposals when the discussion stagnates, means for monitoring the progress of the meeting and encouraging closing, means for organizing important matters extracted during the meeting and generating waiting items, and means for analyzing the previous meeting data and proposing the content of the next meeting. This makes it possible to improve meeting productivity, efficiently manage the progress, and maximize the results of the discussion.

[0620] "Meeting goals" refers to the purpose of the meeting or the objectives to be achieved that are set by the user at the start of the meeting.

[0621] "Real-time analysis" refers to the process of instantly converting voice data into text data and rapidly analyzing its content.

[0622] "Text data" refers to data obtained as a result of converting voice data into text information.

[0623] The "derailment" detection method refers to a function that determines whether the discussion is deviating from the set goal of the meeting and notifies the user.

[0624] An "alert" is a message that notifies the user of important information or a warning.

[0625] "When the discussion stagnates" refers to a situation where the discussion does not progress for a certain period of time during a meeting.

[0626] "Relevant proposals" refer to specific ideas and solutions generated to further the discussion.

[0627] "Means to encourage closing" refers to a function that notifies users when the meeting is about to end and encourages them to wrap up the meeting.

[0628] "Key issues" refer to the most important action items or decisions discussed during the meeting.

[0629] "Queue items" refers to an organized list of action items decided during the meeting.

[0630] "Next meeting content" refers to the topics or agenda items to be discussed at the next meeting based on the previous meeting data.

[0631] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[0632] The system consists of a server and a terminal, and users interact with the server through the terminal. The system utilizes voice recognition technology, natural language processing technology, and generative AI models.

[0633] Hardware and Software Configuration

[0634] The servers are high-performance servers and can utilize the following cloud platforms:

[0635] Amazon Web Services (AWS)

[0636] Google Cloud Platform (GCP)

[0637] The terminals are typical user devices such as PCs, tablets, smartphones, etc. These terminals can access the server via an internet connection.

[0638] The speech recognition software uses Google Speech-to-Text API and IBM Watson Speech to Text to rapidly convert voice data into text data, while Python's Pandas and Numpy are used for data analysis, and NLTK and spaCy are used for automated natural language processing.

[0639] Specific operation of the system

[0640] 1. Setting meeting goals

[0641] The user inputs the goal of the meeting using the terminal, and the terminal sends this information to the server, which stores it in a database.

[0642] Example: A user types, "The goal of today's meeting is to decide on a release schedule for a new product." The server records this.

[0643] 2. Real-time analysis of voice data

[0644] During a meeting, the device collects audio data and streams it to a server in real time. The server converts the audio data into text data and stores the analysis results.

[0645] Example: If a user says, "We are discussing the target customers for a new product," the device collects the speech and the server converts it into text for analysis.

[0646] 3. Derailment detection and alerts

[0647] The server uses the text data to determine whether the discussion is deviating from the goal of the meeting. If so, the server generates an alert and sends it to the device. The device then notifies the user.

[0648] Example: When the conversation shifts to a topic unrelated to the meeting goal, such as "About the weekend events," the server generates an alert such as "Warning: The conversation is straying from the goal" and notifies the device.

[0649] 4. Support for stalled discussions and idea generation

[0650] The server monitors the progress of the meeting and detects impasses when the discussion has stalled for a certain period of time. The server generates relevant proposals and sends them to the terminal, which then presents them to the user.

[0651] Example: When the discussion is "We can't decide on a candidate for a new product name," the server suggests "For example, one name is 'EcoPremium,'" and the terminal displays this.

[0652] 5. Automatic time management and closing

[0653] The terminal monitors the progress of the meeting in real time. When the end time approaches, the terminal sends a notification to the server. The server generates a message prompting the user to close the meeting and sends it to the terminal. The terminal then notifies the user.

[0654] Example: Five minutes before the end of a meeting, the device displays the message, "Meeting will end soon. Please summarize the key points."

[0655] 6. Organizing and automatically generating ToDos

[0656] The server analyzes the data and text collected during the meeting, extracts important action items, and generates a to-do list based on this information and sends it to the device, which is then provided to the user.

[0657] Example: After the meeting ends, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and notifies the device.

[0658] 7. Proposal for the next agenda

[0659] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss in the next meeting. The server generates the next agenda and sends it to the device, which provides it to the user.

[0660] Example: Based on the previous to-do list, the device displays the following on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[0661] Prompt Sentence Examples

[0662] "What is the goal of today's meeting? Example: To decide on a new product release schedule."

[0663] "Determine whether the content of the conversation is aligned with the goal. For example, we are discussing target customers for a new product."

[0664] "Please summarize the key points of the meeting. For example, planning a promotional campaign, gathering feedback on a product prototype, or reviewing marketing strategies."

[0665] As described above, through the embodiment of the present invention, the progress of a meeting can be efficiently managed and specific results can be obtained.

[0666] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0667] Program processing flow

[0668] Step 1: Set meeting goals

[0669] 1. Input: The user inputs the goal of the meeting into the terminal.

[0670] 2. Operation: The device sends the entered goal information to the server, which stores this information in a database and uses it as the basis for subsequent analysis.

[0671] 3. Output: The server stores the goal information, which is the baseline for the entire meeting.

[0672] Specific behavior:

[0673] The user inputs, "The goal of today's meeting is to decide on a release schedule for the new product."

[0674] The terminal sends this information to the server, which stores it in a database.

[0675] Step 2: Real-time analysis of audio data

[0676] 1. Input: The audio data that users say during a meeting.

[0677] 2. Operation: The device collects audio data during the meeting and streams it to the server in real time. The server then uses speech recognition technology to convert the audio data into text data.

[0678] 3. Output: The server stores the converted text data and uses it as the basis for analysis.

[0679] Specific behavior:

[0680] The device collects audio such as, "We are discussing target customers for a new product."

[0681] The server converts this into text and stores the analysis results.

[0682] Step 3: Derailment detection and alerting

[0683] 1. Input: Text data acquired by the server in real time.

[0684] 2. Operation: The server analyzes the text data and determines whether the discussion is in line with the meeting goals. If it detects a deviation, it generates an alert and sends it to the device.

[0685] 3. Output: The device notifies the user of the alert visually or audibly.

[0686] Specific behavior:

[0687] If the conversation turns to a topic unrelated to the goal, such as "about weekend events," the server will determine that the conversation has "digressed."

[0688] The server generates an alert and notifies the device, "Warning: The conversation is straying from the goal."

[0689] Step 4: Resolving stalled discussions and supporting idea generation

[0690] 1. Input: Text data and discussion progress analyzed in real time by the server.

[0691] 2. Operation: The server monitors the progress of the discussion and generates a related proposal if it stalls for a certain period of time. The proposal is then sent to the terminal.

[0692] 3. Output: The device notifies the user of the suggestion.

[0693] Specific behavior:

[0694] When the discussion is "No candidate for a new product name can be decided," the server generates a candidate called "EcoPremium."

[0695] The terminal presents this to the user.

[0696] Step 5: Automatic time management and closing

[0697] 1. Input: The meeting progress time recorded by the device.

[0698] 2. Operation: The terminal monitors the progress of the meeting and notifies the server when the end time approaches. The server generates a closing message and sends it to the terminal.

[0699] 3. Output: The terminal notifies the user with a closing message.

[0700] Specific behavior:

[0701] Five minutes before the end of the meeting, the device will display the message, "The meeting will end soon. Please summarize the main points."

[0702] Step 6: Organize and auto-generate to-dos

[0703] 1. Input: Text data collected during the meeting, action item extraction results.

[0704] 2. Operation: The server analyzes the data, extracts important action items, and generates a to-do list, which is then sent to the device.

[0705] 3. Output: The device notifies the user of the ToDo list.

[0706] Specific behavior:

[0707] After the meeting, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and sends it to the device.

[0708] Step 7: Propose the next agenda

[0709] 1. Input: Previous meeting data and generated to-do list.

[0710] 2. Operation: The server analyzes these data and identifies the topics to be discussed in the next meeting. The generated agenda is sent to the device.

[0711] 3. Output: The terminal notifies the user of the next agenda item.

[0712] Specific behavior:

[0713] Based on the previous to-do list, an agenda for the next meeting is generated: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy, and is displayed on the terminal.

[0714] (Application example 1)

[0715] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0716] Existing assistant systems that improve meeting productivity are designed for use in general office environments, making it difficult to meet the unique needs of specific environments, such as factory production lines. Furthermore, there is a lack of means to efficiently implement functions such as progress management of work instructions, derailment detection, and related idea generation within factories. As a result, work efficiency within factories declines, making productivity improvement a challenge.

[0717] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0718] In this invention, the server includes: means for acquiring meeting goals from a user; means for analyzing voice data in real time and converting it into text data; means for detecting deviations in the meeting based on the text data and sending an alert to the user; means for generating and presenting related ideas when the discussion stalls; means for monitoring the progress of the meeting and encouraging closing; means for organizing action items extracted during the meeting and generating a to-do list; means for analyzing data from the previous meeting and proposing an agenda for the next meeting; means for collecting work instructions in the factory from a user and monitoring the progress of work based on the instructions; and means for generating an alert and notifying workers when work deviates from the plan. This makes it possible to improve work efficiency and productivity even in the factory.

[0719] An "assistance system that improves meeting productivity" is a support system that improves the progress and efficiency of discussions in meetings.

[0720] A "meeting goal" is a target that participants are trying to achieve, set at the start of the meeting.

[0721] "Audio data" refers to data recorded in digital format of conversations during a meeting.

[0722] "Text data" refers to character string information converted from voice data using voice recognition technology.

[0723] "Detecting digressions" means identifying when a discussion deviates from its established goals.

[0724] "Notifying an alert" means notifying the user of a warning by visual or audio means.

[0725] "Generating related ideas" means that if the discussion does not progress, the server automatically generates new proposals and solutions.

[0726] "Monitoring progress time" means recording and managing the time that has elapsed since the start of a meeting or task.

[0727] "Encouraging closure" refers to informing participants that a meeting or task is coming to an end and encouraging them to wrap things up.

[0728] "Action items" refer to important items that were discussed during the meeting or specific next steps to take.

[0729] A "ToDo list" is a list of tasks that need to be done.

[0730] "Factory work instructions" refers to instructions and orders for specific work to be done within a factory.

[0731] "Monitoring work progress" refers to monitoring whether work being done in a factory is progressing as planned.

[0732] "Notifying workers" refers to informing on-site staff involved in the work of important information or warnings.

[0733] The present invention is an assistant system for improving work efficiency in a factory, and a specific embodiment thereof is described below. The system is realized by linking a server and a terminal.

[0734] 1. Setting goals for work instructions

[0735] The server accepts the user's specified goal from the terminal at the start of the task and stores this goal information, which becomes the basis for all subsequent analysis and proposals.

[0736] Example: When a user types into a terminal, "Today's work goal is to check the quality of the new product," the server records this information and uses it as the basis for the entire process.

[0737] 2. Real-time analysis of voice data

[0738] The device transmits the voice data collected during work to the server in real time, and the server converts the voice data into text data using voice recognition technology. At the same time, the server analyzes the text data and determines whether it matches the goal.

[0739] Example: If the audio data collected by the device about a task is, "We are conducting a visual inspection of a new product," the server immediately converts this into text data and analyzes whether the task is in line with the goal.

[0740] 3. Derailment detection and alerts

[0741] If the server determines that the task is deviating from the set goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[0742] Example: If the conversation turns to "About this weekend's events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from goal."

[0743] 4. Support for generating ideas

[0744] The server monitors when the work has not progressed for a certain period of time, detects impasses, and generates related ideas as needed, sending them to the terminal and suggesting them to the user.

[0745] Example: If the problem is "we can't decide on a product name candidate", the server will display a suggestion on the terminal, such as "How about the name 'Quality Plus'?"

[0746] 5. Automatic time management and closing

[0747] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[0748] Example: Five minutes before the scheduled end of a task, the device displays the message, "Your task will be completed soon. Please summarize the main points."

[0749] 6. Organizing and automatically generating ToDos

[0750] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[0751] Example: After the task is completed, the server generates a list such as "Next action items: 1. Collect feedback on the product prototype 2. Propose quality improvements" and displays it on the terminal.

[0752] 7. Proposal for the next agenda

[0753] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, and sends it to the terminal to suggest to the user.

[0754] Example: Based on the previous ToDo list, an agenda item titled "Next work agenda: 1. Evaluate feedback 2. Implement quality improvement measures" is automatically generated and displayed on the device.

[0755] This system uses Python to control the entire program and the "speech_recognition" voice recognition library. The microphones used to collect the voice data are highly sensitive, suitable for use in factories. A specific use case is the issuance of work instructions for product quality checks.

[0756] Example prompt sentence:

[0757] The next work order is:

[0758] 1. Check the quality of all items of new product A

[0759] 2. Update checklist items

[0760] 3. Creation of a report on the check results

[0761] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0762] Step 1:

[0763] The user inputs the task goal into the terminal. The terminal sends this information to the server. The server saves the received goal information. This saved information becomes the basis for all subsequent analysis and suggestions.

[0764] Input: Task goal information entered by the user

[0765] Data processing: The server saves the goal information

[0766] Output: Saved goal information

[0767] Step 2:

[0768] The device collects voice data while working and transmits it in real time to a server, which uses voice recognition technology to convert the voice data into text data, which the server then analyzes to determine whether it meets the goal.

[0769] Input: Audio data collected during work

[0770] Data processing: The server converts the voice data into text data and analyzes the text data.

[0771] Output: Analyzed text data and goal conformance assessment results

[0772] Step 3:

[0773] The server generates a derailment alert based on the converted text data if the task deviates from the goal and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[0774] Input: Text data and goal conformance assessment results

[0775] Data processing: The server generates derailment alerts

[0776] Output: Derailment alert sent to the terminal

[0777] Step 4:

[0778] The server monitors when the work has not progressed for a certain period of time and detects impasses. If necessary, the server generates related ideas, sends them to the terminal, and suggests them to the user.

[0779] Input: Text data and time course information

[0780] Data processing: The server detects impasses and generates ideas

[0781] Output: Related ideas sent to your device

[0782] Step 5:

[0783] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[0784] Input: Start time and expected end time of work

[0785] Data processing: The server generates a closing message

[0786] Output: Closing message sent to the terminal

[0787] Step 6:

[0788] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[0789] Input: Data or text captured during work

[0790] Data processing: The server extracts action items and generates a to-do list

[0791] Output: To-do list sent to the device

[0792] Step 7:

[0793] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, sends it to the terminal, and proposes it to the user.

[0794] Input: Previous work data and generated ToDo list

[0795] Data processing: The server generates the next agenda

[0796] Output: Next agenda sent to terminal

[0797] The above is the specific processing flow of the program of the system that realizes the application example.

[0798] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0799] The present invention relates to an assistant system that improves the productivity of meetings, and furthermore, it combines an emotion engine to recognize and analyze the user's emotional state, thereby more effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[0800] 1. Setting meeting goals

[0801] The server accepts the goals specified by the user from the terminal at the start of the meeting, stores this goal information, and uses this information as the basis for all subsequent analysis and suggestions.

[0802] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as the basis for the entire process.

[0803] 2. Emotion recognition by emotion engine

[0804] The device uses sensors to collect the user's facial expressions and tone of voice during the meeting. This data is sent in real time to the server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[0805] Example: If the user is recognized as feeling stressed, the emotion engine analyzes the information and sends it to the server.

[0806] 3. Real-time analysis of audio data

[0807] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the data into text data for analysis.

[0808] Example: If the meeting audio data collected by the device includes the sentence, "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes it.

[0809] 4. Derailment detection and alerts

[0810] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data, and if the topic deviates from the goal, it generates a derailment alert and sends it to the terminal. The terminal notifies the user of this alert.

[0811] Example: If the conversation veers off topic, such as "about weekend events," the device will display a notification saying, "Warning: Conversation is drifting away from goal."

[0812] 5. Support for generating ideas

[0813] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[0814] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about 'EcoPremium'?"

[0815] 6. Automatic time management and closing

[0816] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[0817] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[0818] 7. Organizing and automatically generating ToDos

[0819] The server extracts important action items discussed during the meeting based on the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[0820] Example: After the meeting ends, the server generates "Next Tasks: 1. Promotion campaign plan 2. Collect feedback on product prototype 3. Review marketing strategy" and displays it on the terminal.

[0821] 8. Proposal for the next agenda

[0822] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[0823] Example: Based on the previous ToDo list, the following agenda is automatically generated and displayed on the device: "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[0824] 9. Moderating discussions based on emotional states

[0825] The server adjusts the progress of the discussion based on the user's emotional state and provides relaxation and motivational suggestions as needed. For example, if the server detects that the user is feeling stressed, it will provide relaxation suggestions.

[0826] Example: If a user is feeling stressed, the server displays a suggestion on the device such as "Take a deep breath and refresh yourself."

[0827] As described above, through the embodiment of the present invention, the progress of a meeting is efficiently managed and appropriate feedback based on the emotion engine is provided, thereby making it possible to achieve concrete results.

[0828] The processing flow will be explained below.

[0829] Step 1:

[0830] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[0831] Step 2:

[0832] The terminal sends the entered goal information to the server, which stores the goal information and organizes it for use in subsequent processes, so that the progress of the entire meeting can be adjusted based on the goal.

[0833] Step 3:

[0834] During meetings, the device uses a microphone and camera to collect voice and facial expression data, which is then streamed to a server in real time.

[0835] Step 4:

[0836] The server converts the received voice data into text data using voice recognition technology, while simultaneously analyzing facial expression data and voice tone to recognize the user's emotional state.

[0837] Step 5:

[0838] The server evaluates whether the discussion is in line with the meeting's goals based on the analyzed text data. If the topic strays from the goal, it generates a derailment alert. It also takes into account the user's emotional state, and generates a message suggesting relaxation if, for example, stress levels are high.

[0839] Step 6:

[0840] The server sends the generated alert and suggestion messages to the terminal, which notifies the user of these messages visually or audibly.

[0841] Example: If the discussion veers off course to "weekend events" and the user is feeling stressed, the device will display "Warning: The conversation is drifting away from the goal. Take a deep breath and refresh yourself."

[0842] Step 7:

[0843] The server monitors when the discussion has not progressed for a certain period of time and detects an impasse. When it detects an impasse, it generates related ideas and sends them to the terminal.

[0844] Step 8:

[0845] The terminal presents the generated ideas to the user, helping the user to restart the discussion based on the new ideas.

[0846] Example: When a new product name cannot be decided, the server generates a suggestion such as "How about 'EcoPremium'?" and the terminal displays this to the user.

[0847] Step 9:

[0848] The terminal monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting. The server then generates a message informing the terminal that the meeting is about to end and sends it to the terminal.

[0849] Step 10:

[0850] The terminal notifies the user of a message prompting the user to close the meeting, and the user begins work to summarize the main points of the meeting.

[0851] Example: Five minutes before the end, the device displays, "We're almost done. Let's summarize the main points."

[0852] Step 11:

[0853] The server extracts important action items discussed during the meeting from the text data acquired during the meeting, organizes them into a to-do list, and sends it to the device.

[0854] Step 12:

[0855] The device displays the generated to-do list to the user, helping them organize their work after the meeting.

[0856] Example: After the meeting, the following tasks are displayed: 1. Plan a promotional campaign 2. Collect feedback on product prototypes 3. Review marketing strategy.

[0857] Step 13:

[0858] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal.

[0859] Step 14:

[0860] The terminal will suggest the next meeting agenda to the user, allowing the user to effectively prepare for the next discussion.

[0861] Example: The next agenda is proposed as follows: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion.

[0862] These are the specific processing steps of the meeting assistant system that combines the emotion engine. This system not only efficiently and effectively manages the progress of meetings, but also provides appropriate feedback based on the user's emotional state, enabling concrete results to be achieved.

[0863] Example 2

[0864] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0865] Conventional meeting management systems have the problem of being unable to grasp the progress of the meeting or the user's emotional state in real time and provide effective support based on that information. This poses the problem of meetings easily going off track or discussions stalling, reducing the overall productivity of the meeting. Furthermore, because appropriate suggestions and adjustments are not made based on the user's emotional state, there is a risk that users' stress will increase and their concentration and motivation will decrease.

[0866] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0867] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, and means for recognizing the user's emotional state in real time and adjusting the progress of the discussion based on the emotion. This makes it possible to prevent derailment and stagnation during meetings, support effective idea generation and action item organization, and provide appropriate feedback according to the user's emotional state.

[0868] A "meeting goal" is a specific objective or purpose that you are trying to achieve through the meeting.

[0869] "User" refers to a person who operates this system and participates in meetings.

[0870] "Audio data" refers to digital data of audio collected during a meeting.

[0871] "Text data" is digital data that has been analyzed and converted into text information from audio data.

[0872] A "digression" refers to a topic that strays from the goal of the meeting or a discussion that goes off-target.

[0873] An "alert" is a warning message that notifies the user of derailments or problems detected by the system.

[0874] "Ideas" refer to new proposals or solutions that advance the discussion.

[0875] "Closing" is the process of concluding a meeting and summarizing outcomes and action items.

[0876] "Action items" are items that are discussed during a meeting and turned into concrete actions or plans.

[0877] A "ToDo list" is an organized list of action items decided during a meeting.

[0878] An "agenda" refers to the specific topics or items to be discussed at the next meeting.

[0879] "Emotional state" refers to the user's current psychological state or mood.

[0880] The "means for adjusting the progress of the discussion" is a function that appropriately adjusts the progress and content of the meeting based on the user's emotional state.

[0881] The present invention relates to an assistant system that improves meeting productivity, and provides technical means for recognizing and analyzing the user's emotional state and effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[0882] 1. Setting meeting goals

[0883] The server accepts the meeting goals specified by the user from the terminal at the start of the meeting and stores this goal information, which is used as the basis for all subsequent analysis and suggestions.

[0884] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as a reference for the entire process. This is done so that the server can store the goal information in a database.

[0885] 2. Emotion recognition by emotion engine

[0886] During a meeting, the device captures the user's facial expressions with a camera and collects the user's tone of voice with a microphone. This data is then sent in real time to a server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[0887] Example: The device's camera and microphone are activated to collect data on the user's facial expressions and voice. The server's emotion engine recognizes that the user is feeling stressed and stores this information in a database.

[0888] 3. Real-time analysis of audio data

[0889] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text data for analysis.

[0890] Example: The device's microphone is activated and captures meeting audio such as "We are discussing target customers for a new product." The server converts this into text data and stores it in a database.

[0891] 4. Derailment detection and alerts

[0892] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data. If the topic deviates from the goal, it generates a digression alert and sends it to the device. The device notifies the user of this alert.

[0893] Example: The server analyzes text data and detects deviations such as "About weekend events." The device displays a notification such as "Warning: The conversation is deviating from the goal."

[0894] 5. Support for generating ideas

[0895] The server monitors when the discussion has not progressed for a certain period of time, detects the impasse, generates related ideas, and sends them to the terminal to suggest to the user.

[0896] Example: The server detects when a meeting is stalling and displays suggestions on the terminal, such as "How about 'EcoPremium'?"

[0897] 6. Automatic time management and closing

[0898] The terminal monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, for example, five minutes before the end, and sends it to the terminal. The terminal then notifies the user of this message.

[0899] Example: The device monitors the progress time and displays the message "The meeting will end soon. Please summarize the main points" five minutes before the scheduled end time.

[0900] 7. Organizing and automatically generating ToDos

[0901] The server extracts important action items from the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[0902] Example: The server generates a list from the discussion content, such as "Next To Do: 1. Plan a promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy," and displays it on the device.

[0903] 8. Proposal for the next agenda

[0904] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[0905] Example: The server analyzes the previous data and generates a list such as "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion" and displays it on the terminal.

[0906] 9. Moderating discussions based on emotional states

[0907] The server adjusts the progress of the discussion based on the user's emotional state, providing relaxation and motivational suggestions as needed. For example, if it detects that the user is feeling stressed, it will provide relaxation suggestions.

[0908] Example: Your device might display a suggestion such as "Take a deep breath and refresh yourself."

[0909] As a result, this assistant system achieves the technical features described in (Claim 1) to (Claim 3), efficiently manages the progress of meetings, and provides appropriate feedback based on the user's emotional state, thereby achieving concrete results.

[0910] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0911] Step 1: Set meeting goals

[0912] Input: The user enters the meeting goal into the device.

[0913] Specific behavior: The user enters "Today's goal is to determine the release schedule for the new product" in the text box and clicks the submit button.

[0914] Data processing: The device receives the entered goal information and sends it to the server.

[0915] Output: Goal information is saved on the server.

[0916] Specific operation: The server records the received goal information in the database and uses it as the basis for subsequent processing.

[0917] Step 2: Recognizing user emotions with the emotion engine

[0918] Input: The user's facial expressions and tone of voice are collected via a camera and microphone.

[0919] What it does: The device's camera captures the user's facial expressions, and the microphone collects the tone of their voice.

[0920] Data processing: The device transmits the collected facial expression and voice data to the server in real time, and the server uses an emotion engine to analyze the data and recognize the user's emotional state.

[0921] Output: The user's emotional state is stored on the server.

[0922] Specific operation: The server's emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[0923] Step 3: Real-time analysis of audio data

[0924] Input: The microphone collects audio data during the meeting.

[0925] Specific operation: The device's microphone is activated and recording the speech of meeting participants.

[0926] Data processing: The device sends the collected voice data to a server in real time. The server uses voice recognition technology to convert the voice data into text data for analysis.

[0927] Output: Text data is generated on the server.

[0928] Specific operation: The server generates text data such as "We are discussing target customers for a new product."

[0929] Step 4: Derailment detection and alerting

[0930] Input: The text data described above.

[0931] Specific behavior: The server analyzes the text data and evaluates whether the discussion is in line with the meeting goals.

[0932] Data processing: The server detects derailments based on text data and generates alerts.

[0933] Output: A derailment alert is sent from the server to the device.

[0934] Specific behavior: The device notifies the user with an alert such as "Warning: The conversation is straying from the goal."

[0935] Step 5: Ideation support

[0936] Input: Meeting progress and stall information.

[0937] Specific operation: The server monitors the progress of the meeting and detects when no progress has been made for a certain period of time.

[0938] Data processing: The server generates relevant ideas and sends them to the device.

[0939] Output: The suggestions are displayed in the terminal.

[0940] What it does: Your device will display suggestions like, "How about 'EcoPremium'?"

[0941] Step 6: Automatic time management and closing

[0942] Input: Meeting start time.

[0943] Specific behavior: The device monitors the meeting progress time.

[0944] Data processing: The terminal notifies the server that the scheduled end time is approaching. The server generates a closing message and sends it to the terminal.

[0945] Output: A closing message is printed to the terminal.

[0946] What happens: The device displays the message "Meeting will end soon. Please summarize the key points."

[0947] Step 7: Organizing and automatically generating to-dos

[0948] Input: Action item information captured during the meeting.

[0949] What it does: The server extracts important action items from the meeting discussion.

[0950] Data processing: The server organizes the action items and generates a to-do list.

[0951] Output: The to-do list is sent to the device.

[0952] Specific actions: The device will display "Next To Do: 1. Plan promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy."

[0953] Step 8: Propose the next agenda

[0954] Input: Previous meeting data and generated to-do list.

[0955] What happens: The server analyzes the previous data and identifies the next topic to discuss.

[0956] Data processing: The server generates the next agenda and sends it to the device.

[0957] Output: The agenda is displayed on the terminal.

[0958] Specific operation: The terminal displays "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[0959] Step 9: Adjust the discussion based on your emotional state

[0960] Input: User emotion data.

[0961] Specific behavior: The server continuously monitors the user's emotional state.

[0962] Data processing: The server generates appropriate suggestions based on the emotional state and sends them to the device.

[0963] Output: The sentiment-based suggestions will be displayed on your device.

[0964] What it does: Your device will display a suggestion such as "Take a deep breath and refresh yourself."

[0965] The above is the flow of processing steps for how the program in this system processes and calculates data to generate specific output. At each step, the server and terminals work together to perform specific operations to improve the user's meeting experience.

[0966] (Application example 2)

[0967] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0968] Efficient and productive meetings are essential in modern manufacturing. However, productivity is often hindered by digressions, stalled discussions, and the emotional state of workers during meetings. This can lead to a decline in the overall efficiency of manufacturing operations and failure to achieve production targets. In particular, a lack of appropriate responses due to changes in emotional states is often the cause. Therefore, there is a need for a system that supports efficient meeting conduct and provides optimal feedback by taking into account the emotional state of workers.

[0969] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0970] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, means for recognizing the user's emotional state using an emotion engine and adjusting the discussion in real time, and means for proposing relaxation and motivation improvement based on the emotional state. This prevents derailment and stagnation in the discussion during the meeting and enables optimal responses based on the emotional state of the participants.

[0971] The "means for acquiring the goal of the meeting from the user" is a function for inputting and saving the goal set by the user at the start of the meeting into the server.

[0972] "Means for analyzing voice data in real time and converting it into text data" refers to a technology that converts voice data collected by a microphone during a meeting into text data in real time.

[0973] "Means of detecting derailment in meetings based on text data and notifying users of an alert" is a function that issues a warning to users if the discussion deviates from the goal based on analyzed text data.

[0974] "Means for generating and presenting related ideas when discussions stagnate" is a function that detects situations where discussions are not progressing and provides users with related suggestions and ideas.

[0975] "Means to monitor the progress of the meeting and encourage closing" is a function that monitors the progress of the meeting and encourages the summary of the main points near the end.

[0976] "A means to organize action items extracted during meetings and generate a to-do list" is a function that extracts important tasks discussed during meetings and generates a structured to-do list.

[0977] "A means of analyzing the data from the previous meeting and proposing the agenda for the next meeting" is a function that analyzes the data and to-do list from the previous meeting and automatically proposes the next meeting topic.

[0978] "Means of using an emotion engine to recognize the user's emotional state and adjust the discussion in real time" is a technology that uses data collected through sensors to analyze the user's emotions and instantly adjust the progress of the discussion.

[0979] The "means for suggesting relaxation techniques and motivation improvement based on the emotional state" is a function that suggests relaxation techniques and motivation improvement as needed based on the user's emotional state.

[0980] The present invention is a system for managing the progress of a meeting using a factory robot and providing feedback based on the emotional state of a worker. The system includes the following means.

[0981] 1. Setting meeting goals

[0982] The server receives the goal specified by the user from the terminal at the start of the meeting. This information is stored on the server and used to progress the entire meeting.

[0983] Example: If you set "Today's production target is to manufacture 1,000 units of product A," the server will record this and use it as the basis for the entire process.

[0984] 2. Real-time analysis of voice data

[0985] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text data for analysis.

[0986] Hardware / software used: Microphone, Python, speech_recognition

[0987] Example: When you say, "There's a delay in parts supply. What should we do?" the server converts this into text in real time and records it.

[0988] 3. Derailment detection and alerts

[0989] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data, and if the topic deviates from the goal, it generates a derailment alert and sends it to the terminal. The terminal notifies the user of this alert.

[0990] Example: If the conversation strays to something like "weekend events," the device will notify you, "We're straying from the current topic."

[0991] 4. Support for generating ideas

[0992] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[0993] Example: If no improvements can be found for a production line, the server might suggest, "For example, how about increasing automation?"

[0994] 5. Automatic time management and closing

[0995] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[0996] Example: Five minutes before the end, announce, "The meeting will end soon. Please summarize the main points."

[0997] 6. Organizing and automatically generating ToDos

[0998] The server extracts important action items discussed during the meeting based on the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[0999] Example: After the meeting ends, the server generates "Next ToDo: 1. Review parts supply 2. Adjust production line" and displays it on the terminal.

[1000] 7. Proposal for the next agenda

[1001] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1002] Example: Automatically generate and display "Next agenda: 1. Measures to improve parts supply 2. Trial operation of new production line."

[1003] 8. Emotional state recognition and discussion coordination using an emotion engine

[1004] The device uses sensors to collect the user's facial expressions and tone of voice during the meeting and transmits this data in real time to the server, which then uses an emotion engine to analyze this data and recognize the user's current emotional state.

[1005] Hardware / software used: Webcam, Python, dlib, FER

[1006] Example: If a worker is feeling tired, the emotion engine will recognize this and the server will suggest, "Take a deep breath."

[1007] Example prompt sentence:

[1008] How will you deal with the delay in parts supply?

[1009] (The robot assistant converts this into text and uses emotion recognition to detect that the worker is feeling stressed.)

[1010] Take a deep breath, refresh yourself, and then think of ideas for improving parts supply.

[1011] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1012] Step 1:

[1013] Meeting goal setting

[1014] Input: The meeting goal specified by the user from their device.

[1015] Data processing: The server receives and stores the goal information entered by the user.

[1016] Output: Goal information saved on the server.

[1017] Specific operation: The user enters "Today's production target is to manufacture 1,000 units of product A" into the terminal and presses the send button. The terminal sends this information to the server, which records it.

[1018] Step 2:

[1019] Audio data collection and real-time analysis

[1020] Input: Audio data during the meeting.

[1021] Data processing: The device collects voice data through the microphone and transmits it to the server in real time. The server then converts the data into text using voice recognition technology.

[1022] Output: Meeting audio converted to text.

[1023] How it works: The device uses a microphone to capture audio during a meeting and sends it to the server, which then uses Python and the speech_recognition library to analyze the audio and turn it into text data.

[1024] Step 3:

[1025] Derailment detection and alert notification

[1026] Input: Parsed text data and user-defined goals.

[1027] Data processing: The server evaluates whether the discussion is in line with the meeting goals based on the text data. If the topic deviates from the goals, an alert is generated and sent to the device.

[1028] Output: Alert notification for discussions that deviate from the goal.

[1029] Specific behavior: The server analyzes the text data, and if the discussion strays from the topic of "weekend events," it sends an alert to the device saying "We're straying from the current topic," and the device displays this notification to the user.

[1030] Step 4:

[1031] Support for generating ideas

[1032] Input: A stalled discussion situation.

[1033] Data processing: The server monitors if the discussion has not progressed for a certain period of time and generates related ideas as needed.

[1034] Output: Related ideas presented to the user.

[1035] How it works: If the discussion stalls, the server uses a generative AI model to generate ideas and sends them to the device, which then suggests to the user, "For example, how about increasing automation?"

[1036] Step 5:

[1037] Automatic time management and closing notifications

[1038] Input: Meeting duration.

[1039] Data processing: The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server then generates a message to prompt the device to close the meeting and sends it to the device.

[1040] Output: Notice of closing.

[1041] Specific operation: The device monitors the meeting time and notifies the server 5 minutes before the end. The server sends a message to the device saying "The meeting will end soon. Please summarize the main points." The device then displays this message to the user.

[1042] Step 6:

[1043] Organize and auto-generate to-dos

[1044] Input: Action items discussed during the meeting.

[1045] Data processing: The server extracts important action items from the data acquired during the meeting and generates a to-do list.

[1046] Output: The generated to-do list.

[1047] Specific operation: After the meeting ends, the server generates a list of "Next To-Dos: 1. Review parts supply 2. Adjust production line" and sends it to the terminal. The terminal displays it to the user.

[1048] Step 7:

[1049] Proposal for next agenda

[1050] Input: Previous meeting data and generated to-do list.

[1051] Data processing: The server analyzes the previous meeting data and identifies the topics to be discussed in the next meeting. It generates the next agenda and sends it to the device.

[1052] Output: The proposed next agenda.

[1053] Specific operation: The server analyzes the previous data, generates "Next agenda: 1. Measures to improve parts supply 2. Trial operation of new production line" and sends it to the terminal. The terminal displays it to the user.

[1054] Step 8:

[1055] Emotional state recognition and discussion coordination using an emotion engine

[1056] Input: The user's facial expressions and tone of voice.

[1057] Data processing: The device transmits data collected by sensors to the server in real time, and the server analyzes the data using an emotion engine to recognize the user's emotional state.

[1058] Output: Suggestions and alerts based on the user's emotional state.

[1059] How it works: The device uses a webcam and microphone to capture facial expressions and tone of voice, and sends them to the server. The server then analyzes them using Python, dlib, and FER, and suggests "take a deep breath" if the user is feeling stressed.

[1060] Example prompt sentence:

[1061] How will you deal with the delay in parts supply?

[1062] (The robot assistant converts this into text and uses emotion recognition to detect that the worker is feeling stressed.)

[1063] Take a deep breath, refresh yourself, and then think of ideas for improving parts supply.

[1064] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1065] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1066] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1067] [Third embodiment]

[1068] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1069] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1070] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1071] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1072] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1073] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1074] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1075] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1076] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1077] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1078] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1079] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1080] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[1081] 1. Setting meeting goals

[1082] The server accepts the goals specified by the user from the terminal at the start of the meeting and stores this goal information, which becomes the basis for all subsequent analysis and suggestions.

[1083] Example: When a user types into a terminal, "The goal of today's meeting is to decide on a release schedule for a new product," the server records this information and uses it as the basis for the entire process.

[1084] 2. Real-time analysis of voice data

[1085] The device transmits the voice data collected during the meeting to the server in real time. The server converts the voice data into text data using speech recognition technology. At the same time, the server analyzes the text data and determines whether it meets the goal.

[1086] Example: If the audio data of a meeting collected by a device includes the phrase "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes whether the agenda is in line with the goal.

[1087] 3. Derailment detection and alerts

[1088] If the server determines that the discussion is deviating from the meeting goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the device, which then notifies the user of the alert visually or audibly.

[1089] Example: If the conversation veers off topic, such as "about weekend events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from the goal."

[1090] 4. Support for generating ideas

[1091] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[1092] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about the name 'EcoPremium'?"

[1093] 5. Automatic time management and closing

[1094] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[1095] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[1096] 6. Organizing and automatically generating ToDos

[1097] The server extracts important action items discussed during the meeting based on the data and text collected during the meeting, organizes them systematically, and generates a to-do list. This to-do list is then sent to the device and provided to the user.

[1098] Example: After the meeting ends, the server generates the following action items: 1. Plan a promotional campaign 2. Collect feedback on the product prototype 3. Review the marketing strategy, and displays them on the device.

[1099] 7. Proposal for the next agenda

[1100] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1101] Example: Based on the previous to-do list, an agenda for the next meeting is automatically generated and displayed on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[1102] As described above, through the embodiment of the present invention, the progress of a meeting can be managed efficiently and specific results can be obtained.

[1103] The processing flow will be explained below.

[1104] Step 1:

[1105] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[1106] Step 2:

[1107] The terminal transmits the entered goal information to the server, which stores and organizes this information for use in subsequent processes.

[1108] Step 3:

[1109] The device uses a microphone to collect audio data during the meeting, which is then streamed to a server in real time.

[1110] Step 4:

[1111] The server uses speech recognition technology to convert the received voice data into text data, which is then stored for analysis.

[1112] Step 5:

[1113] The server uses the analyzed text data to assess whether the discussion is aligned with the meeting's goals, and if the topic deviates from the goals, the server generates an alert containing that information.

[1114] Step 6:

[1115] The server sends the generated alert to the terminal, which notifies the user of the alert visually or audibly.

[1116] Step 7:

[1117] The server detects an impasse when a discussion has not progressed for a certain period of time, and when it detects an impasse, it consults a database or past data to generate related ideas.

[1118] Step 8:

[1119] The server sends the generated idea as a text message to the terminal, which notifies the user of the idea.

[1120] Step 9:

[1121] The device monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting.

[1122] Step 10:

[1123] The server generates a message informing the terminal that the termination is imminent and sends it to the terminal, which then notifies the user with a message prompting the user to close.

[1124] Step 11:

[1125] The server extracts important action items discussed during the meeting based on the data collected during the meeting, and organizes the extracted items into a to-do list.

[1126] Step 12:

[1127] The server sends the generated ToDo list to the terminal, which displays the list to the user.

[1128] Step 13:

[1129] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss at the next meeting.

[1130] Step 14:

[1131] The server generates the next agenda and sends it to the terminal, which then proposes the next meeting agenda to the user.

[1132] This is the specific process flow, which allows the progress of the meeting to be managed efficiently and makes it possible to achieve specific results.

[1133] Example 1

[1134] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1135] Current meeting management systems have problems in that they are unable to adequately manage meeting progress or improve discussion productivity. Conventional systems make it difficult for users to monitor the progress of discussions in real time or immediately detect when a topic has gone off track. Furthermore, when a discussion stalls, users are required to find a solution themselves, which takes time and effort. Furthermore, organizing action items after a meeting and setting the next agenda is time-consuming, hindering efficient work.

[1136] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1137] In this invention, the server includes means for acquiring meeting goals from a user, means for analyzing voice data in real time and converting it into text data, means for detecting deviations in the meeting based on the text data and notifying the user of an alert, means for generating and presenting relevant proposals when the discussion stagnates, means for monitoring the progress of the meeting and encouraging closing, means for organizing important matters extracted during the meeting and generating waiting items, and means for analyzing the previous meeting data and proposing the content of the next meeting. This makes it possible to improve meeting productivity, efficiently manage the progress, and maximize the results of the discussion.

[1138] "Meeting goals" refers to the purpose of the meeting or the objectives to be achieved that are set by the user at the start of the meeting.

[1139] "Real-time analysis" refers to the process of instantly converting voice data into text data and rapidly analyzing its content.

[1140] "Text data" refers to data obtained as a result of converting voice data into text information.

[1141] The "derailment" detection method refers to a function that determines whether the discussion is deviating from the set goal of the meeting and notifies the user.

[1142] An "alert" is a message that notifies the user of important information or a warning.

[1143] "When the discussion stagnates" refers to a situation where the discussion does not progress for a certain period of time during a meeting.

[1144] "Relevant proposals" refer to specific ideas and solutions generated to further the discussion.

[1145] "Means to encourage closing" refers to a function that notifies users when the meeting is about to end and encourages them to wrap up the meeting.

[1146] "Key issues" refer to the most important action items or decisions discussed during the meeting.

[1147] "Queue items" refers to an organized list of action items decided during the meeting.

[1148] "Next meeting content" refers to the topics or agenda items to be discussed at the next meeting based on the previous meeting data.

[1149] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[1150] The system consists of a server and a terminal, and users interact with the server through the terminal. The system utilizes voice recognition technology, natural language processing technology, and generative AI models.

[1151] Hardware and Software Configuration

[1152] The servers are high-performance servers and can utilize the following cloud platforms:

[1153] Amazon Web Services (AWS)

[1154] Google Cloud Platform (GCP)

[1155] The terminals are typical user devices such as PCs, tablets, smartphones, etc. These terminals can access the server via an internet connection.

[1156] The speech recognition software uses Google Speech-to-Text API and IBM Watson Speech to Text to rapidly convert voice data into text data, while Python's Pandas and Numpy are used for data analysis, and NLTK and spaCy are used for automated natural language processing.

[1157] Specific operation of the system

[1158] 1. Setting meeting goals

[1159] The user inputs the goal of the meeting using the terminal, and the terminal sends this information to the server, which stores it in a database.

[1160] Example: A user types, "The goal of today's meeting is to decide on a release schedule for a new product." The server records this.

[1161] 2. Real-time analysis of voice data

[1162] During a meeting, the device collects audio data and streams it to a server in real time. The server converts the audio data into text data and stores the analysis results.

[1163] Example: If a user says, "We are discussing the target customers for a new product," the device collects the speech and the server converts it into text for analysis.

[1164] 3. Derailment detection and alerts

[1165] The server uses the text data to determine whether the discussion is deviating from the goal of the meeting. If so, the server generates an alert and sends it to the device. The device then notifies the user.

[1166] Example: When the conversation shifts to a topic unrelated to the meeting goal, such as "About the weekend events," the server generates an alert such as "Warning: The conversation is straying from the goal" and notifies the device.

[1167] 4. Support for stalled discussions and idea generation

[1168] The server monitors the progress of the meeting and detects impasses when the discussion has stalled for a certain period of time. The server generates relevant proposals and sends them to the terminal, which then presents them to the user.

[1169] Example: When the discussion is "We can't decide on a candidate for a new product name," the server suggests "For example, one name is 'EcoPremium,'" and the terminal displays this.

[1170] 5. Automatic time management and closing

[1171] The terminal monitors the progress of the meeting in real time. When the end time approaches, the terminal sends a notification to the server. The server generates a message prompting the user to close the meeting and sends it to the terminal. The terminal then notifies the user.

[1172] Example: Five minutes before the end of a meeting, the device displays the message, "Meeting will end soon. Please summarize the key points."

[1173] 6. Organizing and automatically generating ToDos

[1174] The server analyzes the data and text collected during the meeting, extracts important action items, and generates a to-do list based on this information and sends it to the device, which is then provided to the user.

[1175] Example: After the meeting ends, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and notifies the device.

[1176] 7. Proposal for the next agenda

[1177] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss in the next meeting. The server generates the next agenda and sends it to the device, which provides it to the user.

[1178] Example: Based on the previous to-do list, the device displays the following on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[1179] Prompt Sentence Examples

[1180] "What is the goal of today's meeting? Example: To decide on a new product release schedule."

[1181] "Determine whether the content of the conversation is aligned with the goal. For example, we are discussing target customers for a new product."

[1182] "Please summarize the key points of the meeting. For example, planning a promotional campaign, gathering feedback on a product prototype, or reviewing marketing strategies."

[1183] As described above, through the embodiment of the present invention, the progress of a meeting can be efficiently managed and specific results can be obtained.

[1184] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1185] Program processing flow

[1186] Step 1: Set meeting goals

[1187] 1. Input: The user inputs the goal of the meeting into the terminal.

[1188] 2. Operation: The device sends the entered goal information to the server, which stores this information in a database and uses it as the basis for subsequent analysis.

[1189] 3. Output: The server stores the goal information, which is the baseline for the entire meeting.

[1190] Specific behavior:

[1191] The user inputs, "The goal of today's meeting is to decide on a release schedule for the new product."

[1192] The terminal sends this information to the server, which stores it in a database.

[1193] Step 2: Real-time analysis of audio data

[1194] 1. Input: The audio data that users say during a meeting.

[1195] 2. Operation: The device collects audio data during the meeting and streams it to the server in real time. The server then uses speech recognition technology to convert the audio data into text data.

[1196] 3. Output: The server stores the converted text data and uses it as the basis for analysis.

[1197] Specific behavior:

[1198] The device collects audio such as, "We are discussing target customers for a new product."

[1199] The server converts this into text and stores the analysis results.

[1200] Step 3: Derailment detection and alerting

[1201] 1. Input: Text data acquired by the server in real time.

[1202] 2. Operation: The server analyzes the text data and determines whether the discussion is in line with the meeting goals. If it detects a deviation, it generates an alert and sends it to the device.

[1203] 3. Output: The device notifies the user of the alert visually or audibly.

[1204] Specific behavior:

[1205] If the conversation turns to a topic unrelated to the goal, such as "about weekend events," the server will determine that the conversation has "digressed."

[1206] The server generates an alert and notifies the device, "Warning: The conversation is straying from the goal."

[1207] Step 4: Resolving stalled discussions and supporting idea generation

[1208] 1. Input: Text data and discussion progress analyzed in real time by the server.

[1209] 2. Operation: The server monitors the progress of the discussion and generates a related proposal if it stalls for a certain period of time. The proposal is then sent to the terminal.

[1210] 3. Output: The device notifies the user of the suggestion.

[1211] Specific behavior:

[1212] When the discussion is "No candidate for a new product name can be decided," the server generates a candidate called "EcoPremium."

[1213] The terminal presents this to the user.

[1214] Step 5: Automatic time management and closing

[1215] 1. Input: The meeting progress time recorded by the device.

[1216] 2. Operation: The terminal monitors the progress of the meeting and notifies the server when the end time approaches. The server generates a closing message and sends it to the terminal.

[1217] 3. Output: The terminal notifies the user with a closing message.

[1218] Specific behavior:

[1219] Five minutes before the end of the meeting, the device will display the message, "The meeting will end soon. Please summarize the main points."

[1220] Step 6: Organize and auto-generate to-dos

[1221] 1. Input: Text data collected during the meeting, action item extraction results.

[1222] 2. Operation: The server analyzes the data, extracts important action items, and generates a to-do list, which is then sent to the device.

[1223] 3. Output: The device notifies the user of the ToDo list.

[1224] Specific behavior:

[1225] After the meeting, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and sends it to the device.

[1226] Step 7: Propose the next agenda

[1227] 1. Input: Previous meeting data and generated to-do list.

[1228] 2. Operation: The server analyzes these data and identifies the topics to be discussed in the next meeting. The generated agenda is sent to the device.

[1229] 3. Output: The terminal notifies the user of the next agenda item.

[1230] Specific behavior:

[1231] Based on the previous to-do list, an agenda for the next meeting is generated: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy, and is displayed on the terminal.

[1232] (Application example 1)

[1233] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] Existing assistant systems that improve meeting productivity are designed for use in general office environments, making it difficult to meet the unique needs of specific environments, such as factory production lines. Furthermore, there is a lack of means to efficiently implement functions such as progress management of work instructions, derailment detection, and related idea generation within factories. As a result, work efficiency within factories declines, making productivity improvement a challenge.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1236] In this invention, the server includes: means for acquiring meeting goals from a user; means for analyzing voice data in real time and converting it into text data; means for detecting deviations in the meeting based on the text data and sending an alert to the user; means for generating and presenting related ideas when the discussion stalls; means for monitoring the progress of the meeting and encouraging closing; means for organizing action items extracted during the meeting and generating a to-do list; means for analyzing data from the previous meeting and proposing an agenda for the next meeting; means for collecting work instructions in the factory from a user and monitoring the progress of work based on the instructions; and means for generating an alert and notifying workers when work deviates from the plan. This makes it possible to improve work efficiency and productivity even in the factory.

[1237] An "assistance system that improves meeting productivity" is a support system that improves the progress and efficiency of discussions in meetings.

[1238] A "meeting goal" is a target that participants are trying to achieve, set at the start of the meeting.

[1239] "Audio data" refers to data recorded in digital format of conversations during a meeting.

[1240] "Text data" refers to character string information converted from voice data using voice recognition technology.

[1241] "Detecting digressions" means identifying when a discussion deviates from its established goals.

[1242] "Notifying an alert" means notifying the user of a warning by visual or audio means.

[1243] "Generating related ideas" means that if the discussion does not progress, the server automatically generates new proposals and solutions.

[1244] "Monitoring progress time" means recording and managing the time that has elapsed since the start of a meeting or task.

[1245] "Encouraging closure" refers to informing participants that a meeting or task is coming to an end and encouraging them to wrap things up.

[1246] "Action items" refer to important items that were discussed during the meeting or specific next steps to take.

[1247] A "ToDo list" is a list of tasks that need to be done.

[1248] "Factory work instructions" refers to instructions and orders for specific work to be done within a factory.

[1249] "Monitoring work progress" refers to monitoring whether work being done in a factory is progressing as planned.

[1250] "Notifying workers" refers to informing on-site staff involved in the work of important information or warnings.

[1251] The present invention is an assistant system for improving work efficiency in a factory, and a specific embodiment thereof is described below. The system is realized by linking a server and a terminal.

[1252] 1. Setting goals for work instructions

[1253] The server accepts the user's specified goal from the terminal at the start of the task and stores this goal information, which becomes the basis for all subsequent analysis and proposals.

[1254] Example: When a user types into a terminal, "Today's work goal is to check the quality of the new product," the server records this information and uses it as the basis for the entire process.

[1255] 2. Real-time analysis of voice data

[1256] The device transmits the voice data collected during work to the server in real time, and the server converts the voice data into text data using voice recognition technology. At the same time, the server analyzes the text data and determines whether it matches the goal.

[1257] Example: If the audio data collected by the device about a task is, "We are conducting a visual inspection of a new product," the server immediately converts this into text data and analyzes whether the task is in line with the goal.

[1258] 3. Derailment detection and alerts

[1259] If the server determines that the task is deviating from the set goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[1260] Example: If the conversation turns to "About this weekend's events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from goal."

[1261] 4. Support for generating ideas

[1262] The server monitors when the work has not progressed for a certain period of time, detects impasses, and generates related ideas as needed, sending them to the terminal and suggesting them to the user.

[1263] Example: If the problem is "we can't decide on a product name candidate", the server will display a suggestion on the terminal, such as "How about the name 'Quality Plus'?"

[1264] 5. Automatic time management and closing

[1265] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[1266] Example: Five minutes before the scheduled end of a task, the device displays the message, "Your task will be completed soon. Please summarize the main points."

[1267] 6. Organizing and automatically generating ToDos

[1268] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[1269] Example: After the task is completed, the server generates a list such as "Next action items: 1. Collect feedback on the product prototype 2. Propose quality improvements" and displays it on the terminal.

[1270] 7. Proposal for the next agenda

[1271] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, and sends it to the terminal to suggest to the user.

[1272] Example: Based on the previous ToDo list, an agenda item titled "Next work agenda: 1. Evaluate feedback 2. Implement quality improvement measures" is automatically generated and displayed on the device.

[1273] This system uses Python to control the entire program and the "speech_recognition" voice recognition library. The microphones used to collect the voice data are highly sensitive, suitable for use in factories. A specific use case is the issuance of work instructions for product quality checks.

[1274] Example prompt sentence:

[1275] The next work order is:

[1276] 1. Check the quality of all items of new product A

[1277] 2. Update checklist items

[1278] 3. Creation of a report on the check results

[1279] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1280] Step 1:

[1281] The user inputs the task goal into the terminal. The terminal sends this information to the server. The server saves the received goal information. This saved information becomes the basis for all subsequent analysis and suggestions.

[1282] Input: Task goal information entered by the user

[1283] Data processing: The server saves the goal information

[1284] Output: Saved goal information

[1285] Step 2:

[1286] The device collects voice data while working and transmits it in real time to a server, which uses voice recognition technology to convert the voice data into text data, which the server then analyzes to determine whether it meets the goal.

[1287] Input: Audio data collected during work

[1288] Data processing: The server converts the voice data into text data and analyzes the text data.

[1289] Output: Analyzed text data and goal conformance assessment results

[1290] Step 3:

[1291] The server generates a derailment alert based on the converted text data if the task deviates from the goal and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[1292] Input: Text data and goal conformance assessment results

[1293] Data processing: The server generates derailment alerts

[1294] Output: Derailment alert sent to the terminal

[1295] Step 4:

[1296] The server monitors when the work has not progressed for a certain period of time and detects impasses. If necessary, the server generates related ideas, sends them to the terminal, and suggests them to the user.

[1297] Input: Text data and time course information

[1298] Data processing: The server detects impasses and generates ideas

[1299] Output: Related ideas sent to your device

[1300] Step 5:

[1301] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[1302] Input: Start time and expected end time of work

[1303] Data processing: The server generates a closing message

[1304] Output: Closing message sent to the terminal

[1305] Step 6:

[1306] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[1307] Input: Data or text captured during work

[1308] Data processing: The server extracts action items and generates a to-do list

[1309] Output: To-do list sent to the device

[1310] Step 7:

[1311] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, sends it to the terminal, and proposes it to the user.

[1312] Input: Previous work data and generated ToDo list

[1313] Data processing: The server generates the next agenda

[1314] Output: Next agenda sent to terminal

[1315] The above is the specific processing flow of the program of the system that realizes the application example.

[1316] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1317] The present invention relates to an assistant system that improves the productivity of meetings, and furthermore, it combines an emotion engine to recognize and analyze the user's emotional state, thereby more effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[1318] 1. Setting meeting goals

[1319] The server accepts the goals specified by the user from the terminal at the start of the meeting, stores this goal information, and uses this information as the basis for all subsequent analysis and suggestions.

[1320] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as the basis for the entire process.

[1321] 2. Emotion recognition by emotion engine

[1322] The device uses sensors to collect the user's facial expressions and tone of voice during the meeting. This data is sent in real time to the server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[1323] Example: If the user is recognized as feeling stressed, the emotion engine analyzes the information and sends it to the server.

[1324] 3. Real-time analysis of audio data

[1325] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the data into text data for analysis.

[1326] Example: If the meeting audio data collected by the device includes the sentence, "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes it.

[1327] 4. Derailment detection and alerts

[1328] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data, and if the topic deviates from the goal, it generates a derailment alert and sends it to the terminal. The terminal notifies the user of this alert.

[1329] Example: If the conversation veers off topic, such as "about weekend events," the device will display a notification saying, "Warning: Conversation is drifting away from goal."

[1330] 5. Support for generating ideas

[1331] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[1332] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about 'EcoPremium'?"

[1333] 6. Automatic time management and closing

[1334] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[1335] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[1336] 7. Organizing and automatically generating ToDos

[1337] The server extracts important action items discussed during the meeting based on the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[1338] Example: After the meeting ends, the server generates "Next Tasks: 1. Promotion campaign plan 2. Collect feedback on product prototype 3. Review marketing strategy" and displays it on the terminal.

[1339] 8. Proposal for the next agenda

[1340] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1341] Example: Based on the previous ToDo list, the following agenda is automatically generated and displayed on the device: "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[1342] 9. Moderating discussions based on emotional states

[1343] The server adjusts the progress of the discussion based on the user's emotional state and provides relaxation and motivational suggestions as needed. For example, if the server detects that the user is feeling stressed, it will provide relaxation suggestions.

[1344] Example: If a user is feeling stressed, the server displays a suggestion on the device such as "Take a deep breath and refresh yourself."

[1345] As described above, through the embodiment of the present invention, the progress of a meeting is efficiently managed and appropriate feedback based on the emotion engine is provided, thereby making it possible to achieve concrete results.

[1346] The processing flow will be explained below.

[1347] Step 1:

[1348] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[1349] Step 2:

[1350] The terminal sends the entered goal information to the server, which stores the goal information and organizes it for use in subsequent processes, so that the progress of the entire meeting can be adjusted based on the goal.

[1351] Step 3:

[1352] During meetings, the device uses a microphone and camera to collect voice and facial expression data, which is then streamed to a server in real time.

[1353] Step 4:

[1354] The server converts the received voice data into text data using voice recognition technology, while simultaneously analyzing facial expression data and voice tone to recognize the user's emotional state.

[1355] Step 5:

[1356] The server evaluates whether the discussion is in line with the meeting's goals based on the analyzed text data. If the topic strays from the goal, it generates a derailment alert. It also takes into account the user's emotional state, and generates a message suggesting relaxation if, for example, stress levels are high.

[1357] Step 6:

[1358] The server sends the generated alert and suggestion messages to the terminal, which notifies the user of these messages visually or audibly.

[1359] Example: If the discussion veers off course to "weekend events" and the user is feeling stressed, the device will display "Warning: The conversation is drifting away from the goal. Take a deep breath and refresh yourself."

[1360] Step 7:

[1361] The server monitors when the discussion has not progressed for a certain period of time and detects an impasse. When it detects an impasse, it generates related ideas and sends them to the terminal.

[1362] Step 8:

[1363] The terminal presents the generated ideas to the user, helping the user to restart the discussion based on the new ideas.

[1364] Example: When a new product name cannot be decided, the server generates a suggestion such as "How about 'EcoPremium'?" and the terminal displays this to the user.

[1365] Step 9:

[1366] The terminal monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting. The server then generates a message informing the terminal that the meeting is about to end and sends it to the terminal.

[1367] Step 10:

[1368] The terminal notifies the user of a message prompting the user to close the meeting, and the user begins work to summarize the main points of the meeting.

[1369] Example: Five minutes before the end, the device displays, "We're almost done. Let's summarize the main points."

[1370] Step 11:

[1371] The server extracts important action items discussed during the meeting from the text data acquired during the meeting, organizes them into a to-do list, and sends it to the device.

[1372] Step 12:

[1373] The device displays the generated to-do list to the user, helping them organize their work after the meeting.

[1374] Example: After the meeting, the following tasks are displayed: 1. Plan a promotional campaign 2. Collect feedback on product prototypes 3. Review marketing strategy.

[1375] Step 13:

[1376] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal.

[1377] Step 14:

[1378] The terminal will suggest the next meeting agenda to the user, allowing the user to effectively prepare for the next discussion.

[1379] Example: The next agenda is proposed as follows: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion.

[1380] These are the specific processing steps of the meeting assistant system that combines the emotion engine. This system not only efficiently and effectively manages the progress of meetings, but also provides appropriate feedback based on the user's emotional state, enabling concrete results to be achieved.

[1381] Example 2

[1382] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1383] Conventional meeting management systems have the problem of being unable to grasp the progress of the meeting or the user's emotional state in real time and provide effective support based on that information. This poses the problem of meetings easily going off track or discussions stalling, reducing the overall productivity of the meeting. Furthermore, because appropriate suggestions and adjustments are not made based on the user's emotional state, there is a risk that users' stress will increase and their concentration and motivation will decrease.

[1384] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1385] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, and means for recognizing the user's emotional state in real time and adjusting the progress of the discussion based on the emotion. This makes it possible to prevent derailment and stagnation during meetings, support effective idea generation and action item organization, and provide appropriate feedback according to the user's emotional state.

[1386] A "meeting goal" is a specific objective or purpose that you are trying to achieve through the meeting.

[1387] "User" refers to a person who operates this system and participates in meetings.

[1388] "Audio data" refers to digital data of audio collected during a meeting.

[1389] "Text data" is digital data that has been analyzed and converted into text information from audio data.

[1390] A "digression" refers to a topic that strays from the goal of the meeting or a discussion that goes off-target.

[1391] An "alert" is a warning message that notifies the user of derailments or problems detected by the system.

[1392] "Ideas" refer to new proposals or solutions that advance the discussion.

[1393] "Closing" is the process of concluding a meeting and summarizing outcomes and action items.

[1394] "Action items" are items that are discussed during a meeting and turned into concrete actions or plans.

[1395] A "ToDo list" is an organized list of action items decided during a meeting.

[1396] An "agenda" refers to the specific topics or items to be discussed at the next meeting.

[1397] "Emotional state" refers to the user's current psychological state or mood.

[1398] The "means for adjusting the progress of the discussion" is a function that appropriately adjusts the progress and content of the meeting based on the user's emotional state.

[1399] The present invention relates to an assistant system that improves meeting productivity, and provides technical means for recognizing and analyzing the user's emotional state and effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[1400] 1. Setting meeting goals

[1401] The server accepts the meeting goals specified by the user from the terminal at the start of the meeting and stores this goal information, which is used as the basis for all subsequent analysis and suggestions.

[1402] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as a reference for the entire process. This is done so that the server can store the goal information in a database.

[1403] 2. Emotion recognition by emotion engine

[1404] During a meeting, the device captures the user's facial expressions with a camera and collects the user's tone of voice with a microphone. This data is then sent in real time to a server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[1405] Example: The device's camera and microphone are activated to collect data on the user's facial expressions and voice. The server's emotion engine recognizes that the user is feeling stressed and stores this information in a database.

[1406] 3. Real-time analysis of audio data

[1407] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text data for analysis.

[1408] Example: The device's microphone is activated and captures meeting audio such as "We are discussing target customers for a new product." The server converts this into text data and stores it in a database.

[1409] 4. Derailment detection and alerts

[1410] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data. If the topic deviates from the goal, it generates a digression alert and sends it to the device. The device notifies the user of this alert.

[1411] Example: The server analyzes text data and detects deviations such as "About weekend events." The device displays a notification such as "Warning: The conversation is deviating from the goal."

[1412] 5. Support for generating ideas

[1413] The server monitors when the discussion has not progressed for a certain period of time, detects the impasse, generates related ideas, and sends them to the terminal to suggest to the user.

[1414] Example: The server detects when a meeting is stalling and displays suggestions on the terminal, such as "How about 'EcoPremium'?"

[1415] 6. Automatic time management and closing

[1416] The terminal monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, for example, five minutes before the end, and sends it to the terminal. The terminal then notifies the user of this message.

[1417] Example: The device monitors the progress time and displays the message "The meeting will end soon. Please summarize the main points" five minutes before the scheduled end time.

[1418] 7. Organizing and automatically generating ToDos

[1419] The server extracts important action items from the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[1420] Example: The server generates a list from the discussion content, such as "Next To Do: 1. Plan a promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy," and displays it on the device.

[1421] 8. Proposal for the next agenda

[1422] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1423] Example: The server analyzes the previous data and generates a list such as "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion" and displays it on the terminal.

[1424] 9. Moderating discussions based on emotional states

[1425] The server adjusts the progress of the discussion based on the user's emotional state, providing relaxation and motivational suggestions as needed. For example, if it detects that the user is feeling stressed, it will provide relaxation suggestions.

[1426] Example: Your device might display a suggestion such as "Take a deep breath and refresh yourself."

[1427] As a result, this assistant system achieves the technical features described in (Claim 1) to (Claim 3), efficiently manages the progress of meetings, and provides appropriate feedback based on the user's emotional state, thereby achieving concrete results.

[1428] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1429] Step 1: Set meeting goals

[1430] Input: The user enters the meeting goal into the device.

[1431] Specific behavior: The user enters "Today's goal is to determine the release schedule for the new product" in the text box and clicks the submit button.

[1432] Data processing: The device receives the entered goal information and sends it to the server.

[1433] Output: Goal information is saved on the server.

[1434] Specific operation: The server records the received goal information in the database and uses it as the basis for subsequent processing.

[1435] Step 2: Recognizing user emotions with the emotion engine

[1436] Input: The user's facial expressions and tone of voice are collected via a camera and microphone.

[1437] What it does: The device's camera captures the user's facial expressions, and the microphone collects the tone of their voice.

[1438] Data processing: The device transmits the collected facial expression and voice data to the server in real time, and the server uses an emotion engine to analyze the data and recognize the user's emotional state.

[1439] Output: The user's emotional state is stored on the server.

[1440] Specific operation: The server's emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[1441] Step 3: Real-time analysis of audio data

[1442] Input: The microphone collects audio data during the meeting.

[1443] Specific operation: The device's microphone is activated and recording the speech of meeting participants.

[1444] Data processing: The device sends the collected voice data to a server in real time. The server uses voice recognition technology to convert the voice data into text data for analysis.

[1445] Output: Text data is generated on the server.

[1446] Specific operation: The server generates text data such as "We are discussing target customers for a new product."

[1447] Step 4: Derailment detection and alerting

[1448] Input: The text data described above.

[1449] Specific behavior: The server analyzes the text data and evaluates whether the discussion is in line with the meeting goals.

[1450] Data processing: The server detects derailments based on text data and generates alerts.

[1451] Output: A derailment alert is sent from the server to the device.

[1452] Specific behavior: The device notifies the user with an alert such as "Warning: The conversation is straying from the goal."

[1453] Step 5: Ideation support

[1454] Input: Meeting progress and stall information.

[1455] Specific operation: The server monitors the progress of the meeting and detects when no progress has been made for a certain period of time.

[1456] Data processing: The server generates relevant ideas and sends them to the device.

[1457] Output: The suggestions are displayed in the terminal.

[1458] What it does: Your device will display suggestions like, "How about 'EcoPremium'?"

[1459] Step 6: Automatic time management and closing

[1460] Input: Meeting start time.

[1461] Specific behavior: The device monitors the meeting progress time.

[1462] Data processing: The terminal notifies the server that the scheduled end time is approaching. The server generates a closing message and sends it to the terminal.

[1463] Output: A closing message is printed to the terminal.

[1464] What happens: The device displays the message "Meeting will end soon. Please summarize the key points."

[1465] Step 7: Organizing and automatically generating to-dos

[1466] Input: Action item information captured during the meeting.

[1467] What it does: The server extracts important action items from the meeting discussion.

[1468] Data processing: The server organizes the action items and generates a to-do list.

[1469] Output: The to-do list is sent to the device.

[1470] Specific actions: The device will display "Next To Do: 1. Plan promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy."

[1471] Step 8: Propose the next agenda

[1472] Input: Previous meeting data and generated to-do list.

[1473] What happens: The server analyzes the previous data and identifies the next topic to discuss.

[1474] Data processing: The server generates the next agenda and sends it to the device.

[1475] Output: The agenda is displayed on the terminal.

[1476] Specific operation: The terminal displays "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[1477] Step 9: Adjust the discussion based on your emotional state

[1478] Input: User emotion data.

[1479] Specific behavior: The server continuously monitors the user's emotional state.

[1480] Data processing: The server generates appropriate suggestions based on the emotional state and sends them to the device.

[1481] Output: The sentiment-based suggestions will be displayed on your device.

[1482] What it does: Your device will display a suggestion such as "Take a deep breath and refresh yourself."

[1483] The above is the flow of processing steps for how the program in this system processes and calculates data to generate specific output. At each step, the server and terminals work together to perform specific operations to improve the user's meeting experience.

[1484] (Application example 2)

[1485] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1486] Efficient and productive meetings are essential in modern manufacturing. However, productivity is often hindered by digressions, stalled discussions, and the emotional state of workers during meetings. This can lead to a decline in the overall efficiency of manufacturing operations and failure to achieve production targets. In particular, a lack of appropriate responses due to changes in emotional states is often the cause. Therefore, there is a need for a system that supports efficient meeting conduct and provides optimal feedback by taking into account the emotional state of workers.

[1487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1488] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, means for recognizing the user's emotional state using an emotion engine and adjusting the discussion in real time, and means for proposing relaxation and motivation improvement based on the emotional state. This prevents derailment and stagnation in the discussion during the meeting and enables optimal responses based on the emotional state of the participants.

[1489] The "means for acquiring the goal of the meeting from the user" is a function for inputting and saving the goal set by the user at the start of the meeting into the server.

[1490] "Means for analyzing voice data in real time and converting it into text data" refers to a technology that converts voice data collected by a microphone during a meeting into text data in real time.

[1491] "Means of detecting derailment in meetings based on text data and notifying users of an alert" is a function that issues a warning to users if the discussion deviates from the goal based on analyzed text data.

[1492] "Means for generating and presenting related ideas when discussions stagnate" is a function that detects situations where discussions are not progressing and provides users with related suggestions and ideas.

[1493] "Means to monitor the progress of the meeting and encourage closing" is a function that monitors the progress of the meeting and encourages the summary of the main points near the end.

[1494] "A means to organize action items extracted during meetings and generate a to-do list" is a function that extracts important tasks discussed during meetings and generates a structured to-do list.

[1495] "A means of analyzing the data from the previous meeting and proposing the agenda for the next meeting" is a function that analyzes the data and to-do list from the previous meeting and automatically proposes the next meeting topic.

[1496] "Means of using an emotion engine to recognize the user's emotional state and adjust the discussion in real time" is a technology that uses data collected through sensors to analyze the user's emotions and instantly adjust the progress of the discussion.

[1497] The "means for suggesting relaxation techniques and motivation improvement based on the emotional state" is a function that suggests relaxation techniques and motivation improvement as needed based on the user's emotional state.

[1498] The present invention is a system for managing the progress of a meeting using a factory robot and providing feedback based on the emotional state of a worker. The system includes the following means.

[1499] 1. Setting meeting goals

[1500] The server receives the goal specified by the user from the terminal at the start of the meeting. This information is stored on the server and used to progress the entire meeting.

[1501] Example: If you set "Today's production target is to manufacture 1,000 units of product A," the server will record this and use it as the basis for the entire process.

[1502] 2. Real-time analysis of voice data

[1503] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text data for analysis.

[1504] Hardware / software used: Microphone, Python, speech_recognition

[1505] Example: When you say, "There's a delay in parts supply. What should we do?" the server converts this into text in real time and records it.

[1506] 3. Derailment detection and alerts

[1507] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data, and if the topic deviates from the goal, it generates a derailment alert and sends it to the terminal. The terminal notifies the user of this alert.

[1508] Example: If the conversation strays to something like "weekend events," the device will notify you, "We're straying from the current topic."

[1509] 4. Support for generating ideas

[1510] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[1511] Example: If no improvements can be found for a production line, the server might suggest, "For example, how about increasing automation?"

[1512] 5. Automatic time management and closing

[1513] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[1514] Example: Five minutes before the end, announce, "The meeting will end soon. Please summarize the main points."

[1515] 6. Organizing and automatically generating ToDos

[1516] The server extracts important action items discussed during the meeting based on the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[1517] Example: After the meeting ends, the server generates "Next ToDo: 1. Review parts supply 2. Adjust production line" and displays it on the terminal.

[1518] 7. Proposal for the next agenda

[1519] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1520] Example: Automatically generate and display "Next agenda: 1. Measures to improve parts supply 2. Trial operation of new production line."

[1521] 8. Emotional state recognition and discussion coordination using an emotion engine

[1522] The device uses sensors to collect the user's facial expressions and tone of voice during the meeting and transmits this data in real time to the server, which then uses an emotion engine to analyze this data and recognize the user's current emotional state.

[1523] Hardware / software used: Webcam, Python, dlib, FER

[1524] Example: If a worker is feeling tired, the emotion engine will recognize this and the server will suggest, "Take a deep breath."

[1525] Example prompt sentence:

[1526] How will you deal with the delay in parts supply?

[1527] (The robot assistant converts this into text and uses emotion recognition to detect that the worker is feeling stressed.)

[1528] Take a deep breath, refresh yourself, and then think of ideas for improving parts supply.

[1529] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1530] Step 1:

[1531] Meeting goal setting

[1532] Input: The meeting goal specified by the user from their device.

[1533] Data processing: The server receives and stores the goal information entered by the user.

[1534] Output: Goal information saved on the server.

[1535] Specific operation: The user enters "Today's production target is to manufacture 1,000 units of product A" into the terminal and presses the send button. The terminal sends this information to the server, which records it.

[1536] Step 2:

[1537] Audio data collection and real-time analysis

[1538] Input: Audio data during the meeting.

[1539] Data processing: The device collects voice data through the microphone and transmits it to the server in real time. The server then converts the data into text using voice recognition technology.

[1540] Output: Meeting audio converted to text.

[1541] How it works: The device uses a microphone to capture audio during a meeting and sends it to the server, which then uses Python and the speech_recognition library to analyze the audio and turn it into text data.

[1542] Step 3:

[1543] Derailment detection and alert notification

[1544] Input: Parsed text data and user-defined goals.

[1545] Data processing: The server evaluates whether the discussion is in line with the meeting goals based on the text data. If the topic deviates from the goals, an alert is generated and sent to the device.

[1546] Output: Alert notification for discussions that deviate from the goal.

[1547] Specific behavior: The server analyzes the text data, and if the discussion strays from the topic of "weekend events," it sends an alert to the device saying "We're straying from the current topic," and the device displays this notification to the user.

[1548] Step 4:

[1549] Support for generating ideas

[1550] Input: A stalled discussion situation.

[1551] Data processing: The server monitors if the discussion has not progressed for a certain period of time and generates related ideas as needed.

[1552] Output: Related ideas presented to the user.

[1553] How it works: If the discussion stalls, the server uses a generative AI model to generate ideas and sends them to the device, which then suggests to the user, "For example, how about increasing automation?"

[1554] Step 5:

[1555] Automatic time management and closing notifications

[1556] Input: Meeting duration.

[1557] Data processing: The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server then generates a message to prompt the device to close the meeting and sends it to the device.

[1558] Output: Notice of closing.

[1559] Specific operation: The device monitors the meeting time and notifies the server 5 minutes before the end. The server sends a message to the device saying "The meeting will end soon. Please summarize the main points." The device then displays this message to the user.

[1560] Step 6:

[1561] Organize and auto-generate to-dos

[1562] Input: Action items discussed during the meeting.

[1563] Data processing: The server extracts important action items from the data acquired during the meeting and generates a to-do list.

[1564] Output: The generated to-do list.

[1565] Specific operation: After the meeting ends, the server generates a list of "Next To-Dos: 1. Review parts supply 2. Adjust production line" and sends it to the terminal. The terminal displays it to the user.

[1566] Step 7:

[1567] Proposal for next agenda

[1568] Input: Previous meeting data and generated to-do list.

[1569] Data processing: The server analyzes the previous meeting data and identifies the topics to be discussed in the next meeting. It generates the next agenda and sends it to the device.

[1570] Output: The proposed next agenda.

[1571] Specific operation: The server analyzes the previous data, generates "Next agenda: 1. Measures to improve parts supply 2. Trial operation of new production line" and sends it to the terminal. The terminal displays it to the user.

[1572] Step 8:

[1573] Emotional state recognition and discussion coordination using an emotion engine

[1574] Input: The user's facial expressions and tone of voice.

[1575] Data processing: The device transmits data collected by sensors to the server in real time, and the server analyzes the data using an emotion engine to recognize the user's emotional state.

[1576] Output: Suggestions and alerts based on the user's emotional state.

[1577] How it works: The device uses a webcam and microphone to capture facial expressions and tone of voice, and sends them to the server. The server then analyzes them using Python, dlib, and FER, and suggests "take a deep breath" if the user is feeling stressed.

[1578] Example prompt sentence:

[1579] How will you deal with the delay in parts supply?

[1580] (The robot assistant converts this into text and uses emotion recognition to detect that the worker is feeling stressed.)

[1581] Take a deep breath, refresh yourself, and then think of ideas for improving parts supply.

[1582] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1583] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1584] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1585] [Fourth embodiment]

[1586] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1587] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1588] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1589] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1590] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1591] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1592] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1593] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1594] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1597] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1598] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1599] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[1600] 1. Setting meeting goals

[1601] The server accepts the goals specified by the user from the terminal at the start of the meeting and stores this goal information, which becomes the basis for all subsequent analysis and suggestions.

[1602] Example: When a user types into a terminal, "The goal of today's meeting is to decide on a release schedule for a new product," the server records this information and uses it as the basis for the entire process.

[1603] 2. Real-time analysis of voice data

[1604] The device transmits the voice data collected during the meeting to the server in real time. The server converts the voice data into text data using speech recognition technology. At the same time, the server analyzes the text data and determines whether it meets the goal.

[1605] Example: If the audio data of a meeting collected by a device includes the phrase "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes whether the agenda is in line with the goal.

[1606] 3. Derailment detection and alerts

[1607] If the server determines that the discussion is deviating from the meeting goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the device, which then notifies the user of the alert visually or audibly.

[1608] Example: If the conversation veers off topic, such as "about weekend events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from the goal."

[1609] 4. Support for generating ideas

[1610] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[1611] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about the name 'EcoPremium'?"

[1612] 5. Automatic time management and closing

[1613] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[1614] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[1615] 6. Organizing and automatically generating ToDos

[1616] The server extracts important action items discussed during the meeting based on the data and text collected during the meeting, organizes them systematically, and generates a to-do list. This to-do list is then sent to the device and provided to the user.

[1617] Example: After the meeting ends, the server generates the following action items: 1. Plan a promotional campaign 2. Collect feedback on the product prototype 3. Review the marketing strategy, and displays them on the device.

[1618] 7. Proposal for the next agenda

[1619] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1620] Example: Based on the previous to-do list, an agenda for the next meeting is automatically generated and displayed on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[1621] As described above, through the embodiment of the present invention, the progress of a meeting can be managed efficiently and specific results can be obtained.

[1622] The processing flow will be explained below.

[1623] Step 1:

[1624] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[1625] Step 2:

[1626] The terminal transmits the entered goal information to the server, which stores and organizes this information for use in subsequent processes.

[1627] Step 3:

[1628] The device uses a microphone to collect audio data during the meeting, which is then streamed to a server in real time.

[1629] Step 4:

[1630] The server uses speech recognition technology to convert the received voice data into text data, which is then stored for analysis.

[1631] Step 5:

[1632] The server uses the analyzed text data to assess whether the discussion is aligned with the meeting's goals, and if the topic deviates from the goals, the server generates an alert containing that information.

[1633] Step 6:

[1634] The server sends the generated alert to the terminal, which notifies the user of the alert visually or audibly.

[1635] Step 7:

[1636] The server detects an impasse when a discussion has not progressed for a certain period of time, and when it detects an impasse, it consults a database or past data to generate related ideas.

[1637] Step 8:

[1638] The server sends the generated idea as a text message to the terminal, which notifies the user of the idea.

[1639] Step 9:

[1640] The device monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting.

[1641] Step 10:

[1642] The server generates a message informing the terminal that the termination is imminent and sends it to the terminal, which then notifies the user with a message prompting the user to close.

[1643] Step 11:

[1644] The server extracts important action items discussed during the meeting based on the data collected during the meeting, and organizes the extracted items into a to-do list.

[1645] Step 12:

[1646] The server sends the generated ToDo list to the terminal, which displays the list to the user.

[1647] Step 13:

[1648] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss at the next meeting.

[1649] Step 14:

[1650] The server generates the next agenda and sends it to the terminal, which then proposes the next meeting agenda to the user.

[1651] This is the specific process flow, which allows the progress of the meeting to be managed efficiently and makes it possible to achieve specific results.

[1652] Example 1

[1653] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1654] Current meeting management systems have problems in that they are unable to adequately manage meeting progress or improve discussion productivity. Conventional systems make it difficult for users to monitor the progress of discussions in real time or immediately detect when a topic has gone off track. Furthermore, when a discussion stalls, users are required to find a solution themselves, which takes time and effort. Furthermore, organizing action items after a meeting and setting the next agenda is time-consuming, hindering efficient work.

[1655] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1656] In this invention, the server includes means for acquiring meeting goals from a user, means for analyzing voice data in real time and converting it into text data, means for detecting deviations in the meeting based on the text data and notifying the user of an alert, means for generating and presenting relevant proposals when the discussion stagnates, means for monitoring the progress of the meeting and encouraging closing, means for organizing important matters extracted during the meeting and generating waiting items, and means for analyzing the previous meeting data and proposing the content of the next meeting. This makes it possible to improve meeting productivity, efficiently manage the progress, and maximize the results of the discussion.

[1657] "Meeting goals" refers to the purpose of the meeting or the objectives to be achieved that are set by the user at the start of the meeting.

[1658] "Real-time analysis" refers to the process of instantly converting voice data into text data and rapidly analyzing its content.

[1659] "Text data" refers to data obtained as a result of converting voice data into text information.

[1660] The "derailment" detection method refers to a function that determines whether the discussion is deviating from the set goal of the meeting and notifies the user.

[1661] An "alert" is a message that notifies the user of important information or a warning.

[1662] "When the discussion stagnates" refers to a situation where the discussion does not progress for a certain period of time during a meeting.

[1663] "Relevant proposals" refer to specific ideas and solutions generated to further the discussion.

[1664] "Means to encourage closing" refers to a function that notifies users when the meeting is about to end and encourages them to wrap up the meeting.

[1665] "Key issues" refer to the most important action items or decisions discussed during the meeting.

[1666] "Queue items" refers to an organized list of action items decided during the meeting.

[1667] "Next meeting content" refers to the topics or agenda items to be discussed at the next meeting based on the previous meeting data.

[1668] The present invention relates to an assistant system for improving the productivity of meetings, and specific embodiments thereof are described below.

[1669] The system consists of a server and a terminal, and users interact with the server through the terminal. The system utilizes voice recognition technology, natural language processing technology, and generative AI models.

[1670] Hardware and Software Configuration

[1671] The servers are high-performance servers and can utilize the following cloud platforms:

[1672] Amazon Web Services (AWS)

[1673] Google Cloud Platform (GCP)

[1674] The terminals are typical user devices such as PCs, tablets, smartphones, etc. These terminals can access the server via an internet connection.

[1675] The speech recognition software uses Google Speech-to-Text API and IBM Watson Speech to Text to rapidly convert voice data into text data, while Python's Pandas and Numpy are used for data analysis, and NLTK and spaCy are used for automated natural language processing.

[1676] Specific operation of the system

[1677] 1. Setting meeting goals

[1678] The user inputs the goal of the meeting using the terminal, and the terminal sends this information to the server, which stores it in a database.

[1679] Example: A user types, "The goal of today's meeting is to decide on a release schedule for a new product." The server records this.

[1680] 2. Real-time analysis of voice data

[1681] During a meeting, the device collects audio data and streams it to a server in real time. The server converts the audio data into text data and stores the analysis results.

[1682] Example: If a user says, "We are discussing the target customers for a new product," the device collects the speech and the server converts it into text for analysis.

[1683] 3. Derailment detection and alerts

[1684] The server uses the text data to determine whether the discussion is deviating from the goal of the meeting. If so, the server generates an alert and sends it to the device. The device then notifies the user.

[1685] Example: When the conversation shifts to a topic unrelated to the meeting goal, such as "About the weekend events," the server generates an alert such as "Warning: The conversation is straying from the goal" and notifies the device.

[1686] 4. Support for stalled discussions and idea generation

[1687] The server monitors the progress of the meeting and detects impasses when the discussion has stalled for a certain period of time. The server generates relevant proposals and sends them to the terminal, which then presents them to the user.

[1688] Example: When the discussion is "We can't decide on a candidate for a new product name," the server suggests "For example, one name is 'EcoPremium,'" and the terminal displays this.

[1689] 5. Automatic time management and closing

[1690] The terminal monitors the progress of the meeting in real time. When the end time approaches, the terminal sends a notification to the server. The server generates a message prompting the user to close the meeting and sends it to the terminal. The terminal then notifies the user.

[1691] Example: Five minutes before the end of a meeting, the device displays the message, "Meeting will end soon. Please summarize the key points."

[1692] 6. Organizing and automatically generating ToDos

[1693] The server analyzes the data and text collected during the meeting, extracts important action items, and generates a to-do list based on this information and sends it to the device, which is then provided to the user.

[1694] Example: After the meeting ends, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and notifies the device.

[1695] 7. Proposal for the next agenda

[1696] The server analyzes the previous meeting data and the generated to-do list to identify topics to discuss in the next meeting. The server generates the next agenda and sends it to the device, which provides it to the user.

[1697] Example: Based on the previous to-do list, the device displays the following on the device: "Next meeting agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy."

[1698] Prompt Sentence Examples

[1699] "What is the goal of today's meeting? Example: To decide on a new product release schedule."

[1700] "Determine whether the content of the conversation is aligned with the goal. For example, we are discussing target customers for a new product."

[1701] "Please summarize the key points of the meeting. For example, planning a promotional campaign, gathering feedback on a product prototype, or reviewing marketing strategies."

[1702] As described above, through the embodiment of the present invention, the progress of a meeting can be efficiently managed and specific results can be obtained.

[1703] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1704] Program processing flow

[1705] Step 1: Set meeting goals

[1706] 1. Input: The user inputs the goal of the meeting into the terminal.

[1707] 2. Operation: The device sends the entered goal information to the server, which stores this information in a database and uses it as the basis for subsequent analysis.

[1708] 3. Output: The server stores the goal information, which is the baseline for the entire meeting.

[1709] Specific behavior:

[1710] The user inputs, "The goal of today's meeting is to decide on a release schedule for the new product."

[1711] The terminal sends this information to the server, which stores it in a database.

[1712] Step 2: Real-time analysis of audio data

[1713] 1. Input: The audio data that users say during a meeting.

[1714] 2. Operation: The device collects audio data during the meeting and streams it to the server in real time. The server then uses speech recognition technology to convert the audio data into text data.

[1715] 3. Output: The server stores the converted text data and uses it as the basis for analysis.

[1716] Specific behavior:

[1717] The device collects audio such as, "We are discussing target customers for a new product."

[1718] The server converts this into text and stores the analysis results.

[1719] Step 3: Derailment detection and alerting

[1720] 1. Input: Text data acquired by the server in real time.

[1721] 2. Operation: The server analyzes the text data and determines whether the discussion is in line with the meeting goals. If it detects a deviation, it generates an alert and sends it to the device.

[1722] 3. Output: The device notifies the user of the alert visually or audibly.

[1723] Specific behavior:

[1724] If the conversation turns to a topic unrelated to the goal, such as "about weekend events," the server will determine that the conversation has "digressed."

[1725] The server generates an alert and notifies the device, "Warning: The conversation is straying from the goal."

[1726] Step 4: Resolving stalled discussions and supporting idea generation

[1727] 1. Input: Text data and discussion progress analyzed in real time by the server.

[1728] 2. Operation: The server monitors the progress of the discussion and generates a related proposal if it stalls for a certain period of time. The proposal is then sent to the terminal.

[1729] 3. Output: The device notifies the user of the suggestion.

[1730] Specific behavior:

[1731] When the discussion is "No candidate for a new product name can be decided," the server generates a candidate called "EcoPremium."

[1732] The terminal presents this to the user.

[1733] Step 5: Automatic time management and closing

[1734] 1. Input: The meeting progress time recorded by the device.

[1735] 2. Operation: The terminal monitors the progress of the meeting and notifies the server when the end time approaches. The server generates a closing message and sends it to the terminal.

[1736] 3. Output: The terminal notifies the user with a closing message.

[1737] Specific behavior:

[1738] Five minutes before the end of the meeting, the device will display the message, "The meeting will end soon. Please summarize the main points."

[1739] Step 6: Organize and auto-generate to-dos

[1740] 1. Input: Text data collected during the meeting, action item extraction results.

[1741] 2. Operation: The server analyzes the data, extracts important action items, and generates a to-do list, which is then sent to the device.

[1742] 3. Output: The device notifies the user of the ToDo list.

[1743] Specific behavior:

[1744] After the meeting, the server generates a to-do list such as "Next action items: 1. Plan a promotional campaign 2. Collect feedback on product prototype 3. Review marketing strategy" and sends it to the device.

[1745] Step 7: Propose the next agenda

[1746] 1. Input: Previous meeting data and generated to-do list.

[1747] 2. Operation: The server analyzes these data and identifies the topics to be discussed in the next meeting. The generated agenda is sent to the device.

[1748] 3. Output: The terminal notifies the user of the next agenda item.

[1749] Specific behavior:

[1750] Based on the previous to-do list, an agenda for the next meeting is generated: 1. Promotion campaign progress report 2. Product prototype feedback review 3. Discussion of new marketing strategy, and is displayed on the terminal.

[1751] (Application example 1)

[1752] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1753] Existing assistant systems that improve meeting productivity are designed for use in general office environments, making it difficult to meet the unique needs of specific environments, such as factory production lines. Furthermore, there is a lack of means to efficiently implement functions such as progress management of work instructions, derailment detection, and related idea generation within factories. As a result, work efficiency within factories declines, making productivity improvement a challenge.

[1754] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1755] In this invention, the server includes: means for acquiring meeting goals from a user; means for analyzing voice data in real time and converting it into text data; means for detecting deviations in the meeting based on the text data and sending an alert to the user; means for generating and presenting related ideas when the discussion stalls; means for monitoring the progress of the meeting and encouraging closing; means for organizing action items extracted during the meeting and generating a to-do list; means for analyzing data from the previous meeting and proposing an agenda for the next meeting; means for collecting work instructions in the factory from a user and monitoring the progress of work based on the instructions; and means for generating an alert and notifying workers when work deviates from the plan. This makes it possible to improve work efficiency and productivity even in the factory.

[1756] An "assistance system that improves meeting productivity" is a support system that improves the progress and efficiency of discussions in meetings.

[1757] A "meeting goal" is a target that participants are trying to achieve, set at the start of the meeting.

[1758] "Audio data" refers to data recorded in digital format of conversations during a meeting.

[1759] "Text data" refers to character string information converted from voice data using voice recognition technology.

[1760] "Detecting digressions" means identifying when a discussion deviates from its established goals.

[1761] "Notifying an alert" means notifying the user of a warning by visual or audio means.

[1762] "Generating related ideas" means that if the discussion does not progress, the server automatically generates new proposals and solutions.

[1763] "Monitoring progress time" means recording and managing the time that has elapsed since the start of a meeting or task.

[1764] "Encouraging closure" refers to informing participants that a meeting or task is coming to an end and encouraging them to wrap things up.

[1765] "Action items" refer to important items that were discussed during the meeting or specific next steps to take.

[1766] A "ToDo list" is a list of tasks that need to be done.

[1767] "Factory work instructions" refers to instructions and orders for specific work to be done within a factory.

[1768] "Monitoring work progress" refers to monitoring whether work being done in a factory is progressing as planned.

[1769] "Notifying workers" refers to informing on-site staff involved in the work of important information or warnings.

[1770] The present invention is an assistant system for improving work efficiency in a factory, and a specific embodiment thereof is described below. The system is realized by linking a server and a terminal.

[1771] 1. Setting goals for work instructions

[1772] The server accepts the user's specified goal from the terminal at the start of the task and stores this goal information, which becomes the basis for all subsequent analysis and proposals.

[1773] Example: When a user types into a terminal, "Today's work goal is to check the quality of the new product," the server records this information and uses it as the basis for the entire process.

[1774] 2. Real-time analysis of voice data

[1775] The device transmits the voice data collected during work to the server in real time, and the server converts the voice data into text data using voice recognition technology. At the same time, the server analyzes the text data and determines whether it matches the goal.

[1776] Example: If the audio data collected by the device about a task is, "We are conducting a visual inspection of a new product," the server immediately converts this into text data and analyzes whether the task is in line with the goal.

[1777] 3. Derailment detection and alerts

[1778] If the server determines that the task is deviating from the set goal based on the text data converted from the voice data, it generates a derailment alert and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[1779] Example: If the conversation turns to "About this weekend's events," the device will display a pop-up notification saying, "Warning: Conversation is drifting away from goal."

[1780] 4. Support for generating ideas

[1781] The server monitors when the work has not progressed for a certain period of time, detects impasses, and generates related ideas as needed, sending them to the terminal and suggesting them to the user.

[1782] Example: If the problem is "we can't decide on a product name candidate", the server will display a suggestion on the terminal, such as "How about the name 'Quality Plus'?"

[1783] 5. Automatic time management and closing

[1784] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[1785] Example: Five minutes before the scheduled end of a task, the device displays the message, "Your task will be completed soon. Please summarize the main points."

[1786] 6. Organizing and automatically generating ToDos

[1787] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[1788] Example: After the task is completed, the server generates a list such as "Next action items: 1. Collect feedback on the product prototype 2. Propose quality improvements" and displays it on the terminal.

[1789] 7. Proposal for the next agenda

[1790] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, and sends it to the terminal to suggest to the user.

[1791] Example: Based on the previous ToDo list, an agenda item titled "Next work agenda: 1. Evaluate feedback 2. Implement quality improvement measures" is automatically generated and displayed on the device.

[1792] This system uses Python to control the entire program and the "speech_recognition" voice recognition library. The microphones used to collect the voice data are highly sensitive, suitable for use in factories. A specific use case is the issuance of work instructions for product quality checks.

[1793] Example prompt sentence:

[1794] The next work order is:

[1795] 1. Check the quality of all items of new product A

[1796] 2. Update checklist items

[1797] 3. Creation of a report on the check results

[1798] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1799] Step 1:

[1800] The user inputs the task goal into the terminal. The terminal sends this information to the server. The server saves the received goal information. This saved information becomes the basis for all subsequent analysis and suggestions.

[1801] Input: Task goal information entered by the user

[1802] Data processing: The server saves the goal information

[1803] Output: Saved goal information

[1804] Step 2:

[1805] The device collects voice data while working and transmits it in real time to a server, which uses voice recognition technology to convert the voice data into text data, which the server then analyzes to determine whether it meets the goal.

[1806] Input: Audio data collected during work

[1807] Data processing: The server converts the voice data into text data and analyzes the text data.

[1808] Output: Analyzed text data and goal conformance assessment results

[1809] Step 3:

[1810] The server generates a derailment alert based on the converted text data if the task deviates from the goal and sends it to the terminal, which then notifies the user of the alert visually or audibly.

[1811] Input: Text data and goal conformance assessment results

[1812] Data processing: The server generates derailment alerts

[1813] Output: Derailment alert sent to the terminal

[1814] Step 4:

[1815] The server monitors when the work has not progressed for a certain period of time and detects impasses. If necessary, the server generates related ideas, sends them to the terminal, and suggests them to the user.

[1816] Input: Text data and time course information

[1817] Data processing: The server detects impasses and generates ideas

[1818] Output: Related ideas sent to your device

[1819] Step 5:

[1820] The terminal monitors the progress of the work and notifies the server when the work is nearing completion. The server generates a message prompting the user to close the work, such as 5 minutes before completion, and sends it to the terminal. The terminal then notifies the user of this message.

[1821] Input: Start time and expected end time of work

[1822] Data processing: The server generates a closing message

[1823] Output: Closing message sent to the terminal

[1824] Step 6:

[1825] The server extracts important action items discussed based on data and text acquired during the work, systematically organizes them, and generates a to-do list, which is then sent to the device and provided to the user.

[1826] Input: Data or text captured during work

[1827] Data processing: The server extracts action items and generates a to-do list

[1828] Output: To-do list sent to the device

[1829] Step 7:

[1830] The server analyzes the previous task data and the generated to-do list to identify topics to tackle in the next task, generates the next agenda, sends it to the terminal, and proposes it to the user.

[1831] Input: Previous work data and generated ToDo list

[1832] Data processing: The server generates the next agenda

[1833] Output: Next agenda sent to terminal

[1834] The above is the specific processing flow of the program of the system that realizes the application example.

[1835] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1836] The present invention relates to an assistant system that improves the productivity of meetings, and furthermore, it combines an emotion engine to recognize and analyze the user's emotional state, thereby more effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[1837] 1. Setting meeting goals

[1838] The server accepts the goals specified by the user from the terminal at the start of the meeting, stores this goal information, and uses this information as the basis for all subsequent analysis and suggestions.

[1839] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as the basis for the entire process.

[1840] 2. Emotion recognition by emotion engine

[1841] The device uses sensors to collect the user's facial expressions and tone of voice during the meeting. This data is sent in real time to the server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[1842] Example: If the user is recognized as feeling stressed, the emotion engine analyzes the information and sends it to the server.

[1843] 3. Real-time analysis of audio data

[1844] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the data into text data for analysis.

[1845] Example: If the meeting audio data collected by the device includes the sentence, "We are discussing target customers for a new product," the server immediately converts this into text data and analyzes it.

[1846] 4. Derailment detection and alerts

[1847] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data, and if the topic deviates from the goal, it generates a derailment alert and sends it to the terminal. The terminal notifies the user of this alert.

[1848] Example: If the conversation veers off topic, such as "about weekend events," the device will display a notification saying, "Warning: Conversation is drifting away from goal."

[1849] 5. Support for generating ideas

[1850] The server monitors when the discussion has not progressed for a certain period of time, detects when it has reached a deadlock, and generates related ideas as needed, sending them to the terminals to suggest to the users.

[1851] Example: If the problem is "We can't decide on a new product name," the server will display a suggestion on the terminal, such as "How about 'EcoPremium'?"

[1852] 6. Automatic time management and closing

[1853] The device monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, such as 5 minutes before the end, and sends it to the device. The device then notifies the user of this message.

[1854] Example: Five minutes before the meeting is scheduled to end, the device displays the message, "Meeting will end soon. Please summarize the key points."

[1855] 7. Organizing and automatically generating ToDos

[1856] The server extracts important action items discussed during the meeting based on the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[1857] Example: After the meeting ends, the server generates "Next Tasks: 1. Promotion campaign plan 2. Collect feedback on product prototype 3. Review marketing strategy" and displays it on the terminal.

[1858] 8. Proposal for the next agenda

[1859] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1860] Example: Based on the previous ToDo list, the following agenda is automatically generated and displayed on the device: "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[1861] 9. Moderating discussions based on emotional states

[1862] The server adjusts the progress of the discussion based on the user's emotional state and provides relaxation and motivational suggestions as needed. For example, if the server detects that the user is feeling stressed, it will provide relaxation suggestions.

[1863] Example: If a user is feeling stressed, the server displays a suggestion on the device such as "Take a deep breath and refresh yourself."

[1864] As described above, through the embodiment of the present invention, the progress of a meeting is efficiently managed and appropriate feedback based on the emotion engine is provided, thereby making it possible to achieve concrete results.

[1865] The processing flow will be explained below.

[1866] Step 1:

[1867] The terminal provides the user with an interface for inputting goals at the start of a meeting. The user inputs the goals of the meeting in a designated field and presses the send button.

[1868] Step 2:

[1869] The terminal sends the entered goal information to the server, which stores the goal information and organizes it for use in subsequent processes, so that the progress of the entire meeting can be adjusted based on the goal.

[1870] Step 3:

[1871] During meetings, the device uses a microphone and camera to collect voice and facial expression data, which is then streamed to a server in real time.

[1872] Step 4:

[1873] The server converts the received voice data into text data using voice recognition technology, while simultaneously analyzing facial expression data and voice tone to recognize the user's emotional state.

[1874] Step 5:

[1875] The server evaluates whether the discussion is in line with the meeting's goals based on the analyzed text data. If the topic strays from the goal, it generates a derailment alert. It also takes into account the user's emotional state, and generates a message suggesting relaxation if, for example, stress levels are high.

[1876] Step 6:

[1877] The server sends the generated alert and suggestion messages to the terminal, which notifies the user of these messages visually or audibly.

[1878] Example: If the discussion veers off course to "weekend events" and the user is feeling stressed, the device will display "Warning: The conversation is drifting away from the goal. Take a deep breath and refresh yourself."

[1879] Step 7:

[1880] The server monitors when the discussion has not progressed for a certain period of time and detects an impasse. When it detects an impasse, it generates related ideas and sends them to the terminal.

[1881] Step 8:

[1882] The terminal presents the generated ideas to the user, helping the user to restart the discussion based on the new ideas.

[1883] Example: When a new product name cannot be decided, the server generates a suggestion such as "How about 'EcoPremium'?" and the terminal displays this to the user.

[1884] Step 9:

[1885] The terminal monitors the progress of the meeting and notifies the server five minutes before the scheduled end of the meeting. The server then generates a message informing the terminal that the meeting is about to end and sends it to the terminal.

[1886] Step 10:

[1887] The terminal notifies the user of a message prompting the user to close the meeting, and the user begins work to summarize the main points of the meeting.

[1888] Example: Five minutes before the end, the device displays, "We're almost done. Let's summarize the main points."

[1889] Step 11:

[1890] The server extracts important action items discussed during the meeting from the text data acquired during the meeting, organizes them into a to-do list, and sends it to the device.

[1891] Step 12:

[1892] The device displays the generated to-do list to the user, helping them organize their work after the meeting.

[1893] Example: After the meeting, the following tasks are displayed: 1. Plan a promotional campaign 2. Collect feedback on product prototypes 3. Review marketing strategy.

[1894] Step 13:

[1895] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal.

[1896] Step 14:

[1897] The terminal will suggest the next meeting agenda to the user, allowing the user to effectively prepare for the next discussion.

[1898] Example: The next agenda is proposed as follows: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion.

[1899] These are the specific processing steps of the meeting assistant system that combines the emotion engine. This system not only efficiently and effectively manages the progress of meetings, but also provides appropriate feedback based on the user's emotional state, enabling concrete results to be achieved.

[1900] Example 2

[1901] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1902] Conventional meeting management systems have the problem of being unable to grasp the progress of the meeting or the user's emotional state in real time and provide effective support based on that information. This poses the problem of meetings easily going off track or discussions stalling, reducing the overall productivity of the meeting. Furthermore, because appropriate suggestions and adjustments are not made based on the user's emotional state, there is a risk that users' stress will increase and their concentration and motivation will decrease.

[1903] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1904] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, and means for recognizing the user's emotional state in real time and adjusting the progress of the discussion based on the emotion. This makes it possible to prevent derailment and stagnation during meetings, support effective idea generation and action item organization, and provide appropriate feedback according to the user's emotional state.

[1905] A "meeting goal" is a specific objective or purpose that you are trying to achieve through the meeting.

[1906] "User" refers to a person who operates this system and participates in meetings.

[1907] "Audio data" refers to digital data of audio collected during a meeting.

[1908] "Text data" is digital data that has been analyzed and converted into text information from audio data.

[1909] A "digression" refers to a topic that strays from the goal of the meeting or a discussion that goes off-target.

[1910] An "alert" is a warning message that notifies the user of derailments or problems detected by the system.

[1911] "Ideas" refer to new proposals or solutions that advance the discussion.

[1912] "Closing" is the process of concluding a meeting and summarizing outcomes and action items.

[1913] "Action items" are items that are discussed during a meeting and turned into concrete actions or plans.

[1914] A "ToDo list" is an organized list of action items decided during a meeting.

[1915] An "agenda" refers to the specific topics or items to be discussed at the next meeting.

[1916] "Emotional state" refers to the user's current psychological state or mood.

[1917] The "means for adjusting the progress of the discussion" is a function that appropriately adjusts the progress and content of the meeting based on the user's emotional state.

[1918] The present invention relates to an assistant system that improves meeting productivity, and provides technical means for recognizing and analyzing the user's emotional state and effectively supporting the progress of meetings. A specific embodiment of this system will be described.

[1919] 1. Setting meeting goals

[1920] The server accepts the meeting goals specified by the user from the terminal at the start of the meeting and stores this goal information, which is used as the basis for all subsequent analysis and suggestions.

[1921] Example: When a user types "Today's goal is to decide the release schedule for a new product" into a terminal and presses the send button, the server records this and uses it as a reference for the entire process. This is done so that the server can store the goal information in a database.

[1922] 2. Emotion recognition by emotion engine

[1923] During a meeting, the device captures the user's facial expressions with a camera and collects the user's tone of voice with a microphone. This data is then sent in real time to a server, which then uses an emotion engine to analyze the data and recognize the user's current emotional state.

[1924] Example: The device's camera and microphone are activated to collect data on the user's facial expressions and voice. The server's emotion engine recognizes that the user is feeling stressed and stores this information in a database.

[1925] 3. Real-time analysis of audio data

[1926] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text data for analysis.

[1927] Example: The device's microphone is activated and captures meeting audio such as "We are discussing target customers for a new product." The server converts this into text data and stores it in a database.

[1928] 4. Derailment detection and alerts

[1929] The server evaluates whether the discussion is in line with the goal of the meeting based on the text data. If the topic deviates from the goal, it generates a digression alert and sends it to the device. The device notifies the user of this alert.

[1930] Example: The server analyzes text data and detects deviations such as "About weekend events." The device displays a notification such as "Warning: The conversation is deviating from the goal."

[1931] 5. Support for generating ideas

[1932] The server monitors when the discussion has not progressed for a certain period of time, detects the impasse, generates related ideas, and sends them to the terminal to suggest to the user.

[1933] Example: The server detects when a meeting is stalling and displays suggestions on the terminal, such as "How about 'EcoPremium'?"

[1934] 6. Automatic time management and closing

[1935] The terminal monitors the progress of the meeting and notifies the server when the meeting is about to end. The server generates a message prompting the user to close the meeting, for example, five minutes before the end, and sends it to the terminal. The terminal then notifies the user of this message.

[1936] Example: The device monitors the progress time and displays the message "The meeting will end soon. Please summarize the main points" five minutes before the scheduled end time.

[1937] 7. Organizing and automatically generating ToDos

[1938] The server extracts important action items from the data acquired during the meeting, organizes them systematically, and generates a to-do list, which is then sent to the device and provided to the user.

[1939] Example: The server generates a list from the discussion content, such as "Next To Do: 1. Plan a promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy," and displays it on the device.

[1940] 8. Proposal for the next agenda

[1941] The server analyzes the previous meeting data and the generated to-do list to identify topics to be discussed in the next meeting, generates the next agenda, and sends it to the terminal to suggest to the user.

[1942] Example: The server analyzes the previous data and generates a list such as "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion" and displays it on the terminal.

[1943] 9. Moderating discussions based on emotional states

[1944] The server adjusts the progress of the discussion based on the user's emotional state, providing relaxation and motivational suggestions as needed. For example, if it detects that the user is feeling stressed, it will provide relaxation suggestions.

[1945] Example: Your device might display a suggestion such as "Take a deep breath and refresh yourself."

[1946] As a result, this assistant system achieves the technical features described in (Claim 1) to (Claim 3), efficiently manages the progress of meetings, and provides appropriate feedback based on the user's emotional state, thereby achieving concrete results.

[1947] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1948] Step 1: Set meeting goals

[1949] Input: The user enters the meeting goal into the device.

[1950] Specific behavior: The user enters "Today's goal is to determine the release schedule for the new product" in the text box and clicks the submit button.

[1951] Data processing: The device receives the entered goal information and sends it to the server.

[1952] Output: Goal information is saved on the server.

[1953] Specific operation: The server records the received goal information in the database and uses it as the basis for subsequent processing.

[1954] Step 2: Recognizing user emotions with the emotion engine

[1955] Input: The user's facial expressions and tone of voice are collected via a camera and microphone.

[1956] What it does: The device's camera captures the user's facial expressions, and the microphone collects the tone of their voice.

[1957] Data processing: The device transmits the collected facial expression and voice data to the server in real time, and the server uses an emotion engine to analyze the data and recognize the user's emotional state.

[1958] Output: The user's emotional state is stored on the server.

[1959] Specific operation: The server's emotion engine analyzes the user's emotions and recognizes that the user is feeling stressed.

[1960] Step 3: Real-time analysis of audio data

[1961] Input: The microphone collects audio data during the meeting.

[1962] Specific operation: The device's microphone is activated and recording the speech of meeting participants.

[1963] Data processing: The device sends the collected voice data to a server in real time. The server uses voice recognition technology to convert the voice data into text data for analysis.

[1964] Output: Text data is generated on the server.

[1965] Specific operation: The server generates text data such as "We are discussing target customers for a new product."

[1966] Step 4: Derailment detection and alerting

[1967] Input: The text data described above.

[1968] Specific behavior: The server analyzes the text data and evaluates whether the discussion is in line with the meeting goals.

[1969] Data processing: The server detects derailments based on text data and generates alerts.

[1970] Output: A derailment alert is sent from the server to the device.

[1971] Specific behavior: The device notifies the user with an alert such as "Warning: The conversation is straying from the goal."

[1972] Step 5: Ideation support

[1973] Input: Meeting progress and stall information.

[1974] Specific operation: The server monitors the progress of the meeting and detects when no progress has been made for a certain period of time.

[1975] Data processing: The server generates relevant ideas and sends them to the device.

[1976] Output: The suggestions are displayed in the terminal.

[1977] What it does: Your device will display suggestions like, "How about 'EcoPremium'?"

[1978] Step 6: Automatic time management and closing

[1979] Input: Meeting start time.

[1980] Specific behavior: The device monitors the meeting progress time.

[1981] Data processing: The terminal notifies the server that the scheduled end time is approaching. The server generates a closing message and sends it to the terminal.

[1982] Output: A closing message is printed to the terminal.

[1983] What happens: The device displays the message "Meeting will end soon. Please summarize the key points."

[1984] Step 7: Organizing and automatically generating to-dos

[1985] Input: Action item information captured during the meeting.

[1986] What it does: The server extracts important action items from the meeting discussion.

[1987] Data processing: The server organizes the action items and generates a to-do list.

[1988] Output: The to-do list is sent to the device.

[1989] Specific actions: The device will display "Next To Do: 1. Plan promotion campaign 2. Collect feedback on product prototype 3. Review marketing strategy."

[1990] Step 8: Propose the next agenda

[1991] Input: Previous meeting data and generated to-do list.

[1992] What happens: The server analyzes the previous data and identifies the next topic to discuss.

[1993] Data processing: The server generates the next agenda and sends it to the device.

[1994] Output: The agenda is displayed on the terminal.

[1995] Specific operation: The terminal displays "Next agenda: 1. Promotion campaign progress report 2. Product prototype feedback review 3. New marketing strategy discussion."

[1996] Step 9: Adjust the discussion based on your emotional state

[1997] Input: User emotion data.

[1998] Specific behavior: The server continuously monitors the user's emotional state.

[1999] Data processing: The server generates appropriate suggestions based on the emotional state and sends them to the device.

[2000] Output: The sentiment-based suggestions will be displayed on your device.

[2001] What it does: Your device will display a suggestion such as "Take a deep breath and refresh yourself."

[2002] The above is the flow of processing steps for how the program in this system processes and calculates data to generate specific output. At each step, the server and terminals work together to perform specific operations to improve the user's meeting experience.

[2003] (Application example 2)

[2004] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2005] Efficient and productive meetings are essential in modern manufacturing. However, productivity is often hindered by digressions, stalled discussions, and the emotional state of workers during meetings. This can lead to a decline in the overall efficiency of manufacturing operations and failure to achieve production targets. In particular, a lack of appropriate responses due to changes in emotional states is often the cause. Therefore, there is a need for a system that supports efficient meeting conduct and provides optimal feedback by taking into account the emotional state of workers.

[2006] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2007] In this invention, the server includes means for acquiring the meeting goal from the user, means for analyzing voice data in real time and converting it into text data, means for detecting derailment in the meeting based on the text data and notifying the user of an alert, means for generating and presenting related ideas when the discussion stagnates, means for monitoring the meeting progress time and encouraging closing, means for organizing action items extracted during the meeting and generating a to-do list, means for analyzing the previous meeting data and proposing the next meeting agenda, means for recognizing the user's emotional state using an emotion engine and adjusting the discussion in real time, and means for proposing relaxation and motivation improvement based on the emotional state. This prevents derailment and stagnation in the discussion during the meeting and enables optimal responses based on the emotional state of the participants.

[2008] The "means for acquiring the goal of the meeting from the user" is a function for inputting and saving the goal set by the user at the start of the meeting into the server.

[2009] "Means for analyzing voice data in real time and converting it into text data" refers to a technology that converts voice data collected by a microphone during a meeting into text data in real time.

[2010] "Means of detecting derailment in meetings based on text data and notifying users of an alert" is a function that issues a warning to users if the discussion deviates from the goal based on analyzed text data.

[2011] "Means for generating and presenting related ideas when discussions stagnate" is a function that detects situations where discussions are not progressing and provides users with related suggestions and ideas.

[2012] "Means to monitor the progress of the meeting and encourage closing" is a function that monitors the progress of the meeting and encourages the summary of the main points near the end.

[2013] "A means to organize action items extracted during meetings and generate a to-do list" is a function that extracts important tasks discussed during meetings and generates a structured to-do list.

[2014] "A means of analyzing the data from the previous meeting and proposing the agenda for the next meeting" is a function that analyzes the data and to-do list from the previous meeting and automatically proposes the next meeting topic.

[2015] "Means of using an emotion engine to recognize the user's emotional state and adjust the discussion in real time" is a technology that uses data collected through sensors to analyze the user's emotions and instantly adjust the progress of the discussion.

[2016] The "means for suggesting relaxation techniques and motivation improvement based on the emotional state" is a function that suggests relaxation techniques and motivation improvement as needed based on the user's emotional state.

[2017] The present invention is a system for managing the progress of a meeting using a factory robot and providing feedback based on the emotional state of a worker. The system includes the following means.

[2018] 1. Setting meeting goals

[2019] The server receives the goal specified by the user from the terminal at the start of the meeting. This information is stored on the server and used to progress the entire meeting.

[2020] Example: If you set "Today's production target is to manufacture 1,000 units of product A," the server will record this and use it as the basis for the entire process.

[2021] 2. Real-time analysis of voice data

[2022] The device collects audio data during the meeting through a microphone and transmits it in real time to a server, which then uses voice recognition technology to convert the audio data into text ...

Claims

1. An assistant system for improving meeting productivity, a means for obtaining a goal of the meeting from the user; A means for analyzing voice data in real time and converting it into text data; A method for detecting derailment in meetings based on text data and notifying users of an alert; a means of generating and presenting relevant ideas when discussions stall; A way to monitor meeting time and encourage closing, A way to organize action items extracted during meetings and generate a to-do list. A means to analyze previous meeting data and propose the next meeting agenda, A system including:

2. 2. The system according to claim 1, further comprising means for detecting a topic that deviates from the goal of the meeting and notifying the user of an alert in real time.

3. 2. The system according to claim 1, further comprising means for analyzing the voice data in real time and presenting related ideas to the user when an impasse is detected.

Images