System

Abstract

A system is provided.SOLUTION: The system includes means for analyzing an input meeting schedule and meeting content and storing meeting details in a database, means for preparing a generation AI based on the stored meeting details, setting roles, and inputting meeting content, means for providing a live feed of the meeting to the generation AI, and for the generation AI to make appropriate questions and supplementary explanations based on the utilization of the live feed, and means for extracting next action items from the meeting log and distributing the next action items to participants.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional conference systems, appropriate questions are often not asked during meetings, and problems are often overlooked. As a result, even if a meeting ends smoothly, it may not be productive. Another issue is that creating meeting minutes and organizing action items for the next meeting takes time and effort, making effective follow-up difficult. The present invention aims to solve these issues and provide a conference support system that utilizes generative AI to improve the effectiveness of meetings. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system that includes a means for analyzing input meeting schedules and meeting contents and saving detailed information about the meeting in a database, a means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting contents, a means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the usage content, a means for collecting log data during the meeting, automatically generating minutes, and distributing them to participants, and a means for extracting next action items from the meeting log data and distributing them to participants.

[0006] In addition, the generation AI can analyze what is being said in real time during the meeting and provide additional questions or explanations as needed, further increasing the effectiveness of the meeting.

[0007] Furthermore, by providing a means to automatically distribute the minutes and action item list generated after the meeting to participants, the efficiency of follow-up can be improved and task management until the next meeting can be simplified.

[0008] A "meeting schedule" is information that includes the date, time, and location of a meeting.

[0009] "Meeting Content" means the topics, agenda, and related details to be discussed at the meeting.

[0010] A "database" is an electronic information storage system for organizing and storing meeting details and log data.

[0011] "Generative AI" is a system that uses artificial intelligence technology to participate in meetings and ask appropriate questions or provide additional explanations.

[0012] A "live feed" is information such as audio, text, and video that occurs in real time as the meeting progresses.

[0013] "Log data" refers to electronic data that records statements, discussions, decisions, etc. that occur during a meeting.

[0014] A "minutes" is a document that summarizes the contents of a meeting, statements made, decisions made, etc.

[0015] "Action items" are tasks or activities that result from a meeting and must be completed before the next meeting.

[0016] A "role" is a specific function or responsibility that a generative AI participating in a meeting will have. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system for improving the effectiveness of meetings by utilizing generative AI, and detailed embodiments thereof will be described below.

[0039] Input of meeting schedule and content

[0040] Users input the meeting schedule and details into the server from their devices (PCs or smartphones). Specifically, they fill out an input form with detailed information such as the meeting date, time, location, topic, agenda, and the role they want the AI ​​to play (e.g., questioner, supplementary explanation), and then click the send button.

[0041] Preparing the Generative AI

[0042] The server analyzes the received meeting details and stores them in a database. It prepares the generative AI based on the stored information and loads the necessary libraries and models to input the meeting content and roles into the AI.

[0043] Generative AI meeting participation

[0044] The server provides a live feed (real-time data of the meeting) to the AI ​​generator. Once the meeting begins, the AI ​​generator analyzes the comments and discussions in real time, asking appropriate questions and providing supplementary explanations.

[0045] Taking meeting minutes

[0046] The server collects log data during the meeting and automatically generates minutes, which include the date and time of the meeting, participants, agenda, comments made, and decisions made. These minutes are distributed to participants after the meeting ends.

[0047] Extracting action items

[0048] The server extracts next action items from the meeting log data. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[0049] Specific examples

[0050] A concrete example is a meeting about a marketing plan for a new product.

[0051] 1. The user enters the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[0052] 2. The server receives this information, initializes the generating AI, and inputs the meeting contents into the AI.

[0053] 3. Once the meeting begins, the generative AI analyzes the live meeting feed and provides additional explanations at the appropriate time.

[0054] 4. The server automatically generates minutes from the log data during the meeting and distributes them to participants after the meeting ends.

[0055] 5. The server extracts next action items from the meeting log data, organizes them into a list, and distributes it to participants.

[0056] In this way, the system of the present invention increases the effectiveness of meetings and simplifies post-meeting follow-up by automating the creation of meeting minutes and organizing action items.

[0057] The processing flow will be explained below.

[0058] Step 1:

[0059] The user enters the meeting schedule and content from a device (PC or smartphone), filling in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and then clicks the submit button.

[0060] Step 2:

[0061] The server parses the incoming data, checking what the user has entered and converting it to the required format, checking for consistency of data format and the presence of all required fields.

[0062] Step 3:

[0063] The server stores the formatted meeting details in a database, including the meeting schedule, content, participant list, and the role of the generating AI.

[0064] Step 4:

[0065] The server initializes the generated AI, loading the necessary libraries and AI models, and preparing the generated AI instance. Memory and CPU / GPU resources are also allocated at this stage.

[0066] Step 5:

[0067] The server inputs detailed information about the meeting into the AI ​​generator, including the meeting topic and assigned roles, and instructs the AI ​​on the next task to be performed.

[0068] Step 6:

[0069] The server streams the live feed (real-time data of the meeting) to the generation AI. After the meeting starts, it provides the generation AI with data such as audio, text, and video that is generated in real time.

[0070] Step 7:

[0071] The generative AI analyzes what is being said in real time during the meeting. As the meeting progresses, the generative AI monitors what is being said and discussed in real time, asking appropriate questions and providing supplementary explanations as needed.

[0072] Step 8:

[0073] The server collects log data during the meeting, recording all statements, discussions, and decisions made in the meeting as logs and saving them for later analysis.

[0074] Step 9:

[0075] The server automatically generates minutes based on the meeting log data. These minutes include the meeting date and time, participants, agenda, comments, and decisions. The generated minutes are then checked for errors and omissions.

[0076] Step 10:

[0077] The server distributes the generated minutes to participants after the meeting ends. The minutes are sent to all participants via email or a dedicated system.

[0078] Step 11:

[0079] The server extracts the next action items from the meeting log data, analyzes the log data, and lists the tasks and follow-up items that need to be done by the next meeting.

[0080] Step 12:

[0081] The server distributes the generated action item list to participants, and the list is sent to all participants via email or a dedicated system, improving follow-up efficiency.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Conventional meeting support systems have problems with insufficient understanding of meeting content and efficient follow-up, resulting in time and effort being required even after the meeting has ended. Furthermore, it is difficult to provide appropriate supplementary explanations and questions in real time based on the content and progress of comments made during the meeting, which means that the effectiveness of the meeting cannot be maximized. Furthermore, follow-up tasks such as creating minutes and organizing action items are often done manually, and there is a need for greater efficiency.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the content of the meeting, means for collecting log data during the meeting, automatically generating minutes and distributing them to participants, means for extracting next action items from the meeting log data and distributing them to participants, means for acquiring real-time data from the conference tool and providing it to the generation AI, means for the generation AI to analyze remarks in real time and generate questions and provide supplementary explanations, and means for distributing the automatically generated minutes and action item list to participants after the meeting, thereby enabling efficient meetings and effective follow-up.

[0087] The "input meeting schedule and meeting content" is data that the user inputs into the server via the terminal, including detailed information such as the start date and time of the meeting, the location, the topic, and the agenda.

[0088] A "database" is a storage device for efficiently storing and managing detailed information about analyzed meetings, log data, and configuration information for the generation AI.

[0089] "Generative AI" is an artificial intelligence model that asks appropriate questions and provides supplementary explanations based on the content of a meeting, and is a system that operates using specific libraries and algorithms.

[0090] "Live feed" refers to the audio and text data generated in real time during a meeting, which the server provides to the generating AI, allowing the generating AI to participate in the meeting.

[0091] "Log data" refers to a record of all statements made during a meeting and other related data that is later used to create minutes and action items.

[0092] Minutes are documents that summarize the main points of a meeting by organizing what was said, decisions made, and agenda items, and are distributed to participants after the meeting has ended.

[0093] "Action items" are tasks and follow-up matters decided during the meeting that must be completed before the next meeting. By extracting and distributing these to participants, it becomes clear what needs to be completed before the next meeting.

[0094] "Real-time data" refers to the audio and text information generated in real time during a meeting, which is then analyzed by the generating AI.

[0095] The present invention is a system that uses generative AI to improve the efficiency and usefulness of meetings. Specific embodiments for carrying out the present invention will be described in detail below.

[0096] Hardware and Software Use

[0097] This system uses a server, a terminal, and a generative AI model. The terminal is a personal computer or smartphone that communicates with the server. The server has powerful computing power and runs the generative AI model. The following specific software is used:

[0098] Database management systems: MySQL or PostgreSQL

[0099] AI model libraries: TensorFlow, PyTorch

[0100] Communication protocol: HTTPS

[0101] Natural language processing technology: Natural language analysis libraries (e.g., spaCy, NLTK)

[0102] Meeting tool: General online meeting tools (e.g., Zoom or Microsoft Teams)

[0103] System Operation

[0104] The user uses the terminal to input the schedule and content of the meeting. For example, in the case of a meeting about a marketing plan for a new product, the user would input the following:

[0105] Meeting Schedule: October 15, 2023, 14:00

[0106] Meeting topic: New product marketing plan

[0107] The role of generative AI: Supplementary explanation

[0108] When the user clicks the submit button, the input data is sent to the server.

[0109] The server analyzes the received meeting information, converts the meeting details into JSON format, and stores it in a database. Next, it loads the libraries and models required to prepare the generative AI model (e.g., a generative AI model using TensorFlow) and configures it according to the content and roles of the meeting.

[0110] When a meeting starts, the server acquires real-time data from the meeting tool. This data includes audio and text information generated during the meeting. The real-time data is provided to the generation AI, which analyzes what is being said. For example, the generation AI generates appropriate questions and supplementary explanations in real time and outputs them as text through the meeting tool.

[0111] The server also collects all log data during the meeting. After the meeting, minutes are automatically generated based on the log data. The generated minutes include the date and time, participants, agenda, comments, and decisions. The minutes are then distributed to participants via email.

[0112] Furthermore, the server extracts action items from the log data, including next follow-up tasks, and distributes them to participants as a list, allowing for efficient management of tasks decided in the meeting.

[0113] In this way, the invention centralizes meeting efficiency and follow-up management, and by using generative AI, it is possible to maximize the usefulness of meetings.

[0114] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0115] Step 1:

[0116] The user inputs the schedule and details of the meeting from the terminal and sends it.

[0117] A user launches a conference management application using a PC or smartphone.

[0118] In the input form, enter the date, time, location, topic, agenda, and desired role of the generating AI for the meeting.

[0119] Click the submit button to send the input data to the server.

[0120] (Input) Meeting schedule, content, and generation AI role

[0121] (Output) Meeting information sent to the server

[0122] Step 2:

[0123] The server analyzes the received conference information and stores it in a database.

[0124] The server converts the received conference information into JSON format.

[0125] Save the JSON format data to the database.

[0126] (Input) Meeting information received from the user

[0127] (Output) Meeting information stored in the database

[0128] Step 3:

[0129] The server prepares the generative AI model.

[0130] The meeting information is obtained from the database and the generating AI is initialized according to the content and role of the meeting.

[0131] Loading a generative AI model (e.g. using TensorFlow) and required libraries.

[0132] (Input) Meeting information stored in the database

[0133] (Output) Initialized generated AI

[0134] Step 4:

[0135] The server collects real-time data from the meeting and provides it to the generation AI.

[0136] When a meeting starts, the server obtains real-time audio and text data from the meeting tool (e.g., Zoom, Microsoft Teams).

[0137] The acquired real-time data is passed to the generation AI.

[0138] (Input) Real-time data from conferencing tools

[0139] (Output) Real-time data provided to the generative AI

[0140] Step 5:

[0141] The generative AI analyzes the content of statements in real time and generates appropriate questions and supplementary explanations.

[0142] The generative AI analyzes real-time data and generates necessary questions and supplementary explanations based on what is being said and the progress of the meeting.

[0143] The generated questions and supplementary explanations are output to the conference tool and displayed to participants in real time.

[0144] (Input) Real-time data and meeting information

[0145] (Output) Generated questions and supplementary explanations

[0146] Step 6:

[0147] The server collects all log data during the meeting and automatically generates minutes.

[0148] Accumulates log data obtained from conference tools.

[0149] Minutes are automatically generated from the accumulated log data using a minutes generation algorithm.

[0150] The generated minutes include the date, time, participants, agenda, what was said, and what was decided.

[0151] (Input) Log data during the meeting

[0152] (Output) Automatically generated minutes

[0153] Step 7:

[0154] The server distributes the minutes and action item list to participants.

[0155] Automatically generated meeting minutes and a list of action items extracted from the log data.

[0156] The generated minutes and action item list are distributed to participants via email or other means.

[0157] (Input) Automatically generated minutes and action items

[0158] (Output) Meeting minutes and action item list distributed to participants

[0159] The above steps will improve the efficiency and usefulness of meetings.

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] In modern manufacturing, the effectiveness of meetings and their follow-up are important for maximizing factory operational efficiency. However, the traditional manual process of creating meeting minutes and extracting action items requires a great deal of time and effort, resulting in a decline in meeting efficiency. Furthermore, it can be difficult to ask questions or provide supplementary explanations in real time, which can sometimes prevent meetings from proceeding smoothly. To solve this problem, a more efficient meeting support system is needed.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the utilization content, and means for a robot located in the factory to monitor the meeting and analyze the progress of the meeting in real time to provide assistance. This increases the effectiveness of meetings and enables the automatic generation of minutes and the extraction and distribution of action items.

[0165] "Meeting schedule" refers to information that includes the date, time, location and associated timeline of a meeting.

[0166] "Meeting content" refers to information including the agenda, topics, and participant roles in a meeting.

[0167] A "database" is an information management system that stores conference details and allows them to be retrieved at a later time.

[0168] "Generative AI" is a system that uses artificial intelligence technology to automatically generate questions and supplementary explanations based on the content of the meeting.

[0169] "Live feed" refers to streaming data that sends real-time information about the progress of a meeting to the generation AI.

[0170] "Questions" are posed by the generative AI in response to comments made or topics discussed during a meeting, and are intended to deepen the discussion.

[0171] "Supplementary explanation" refers to additional information or explanations that make the content of the meeting easier to understand.

[0172] "Log data" refers to digital data that records what was said during a meeting and the progress of the discussion.

[0173] "Minutes" are official documents summarizing the proceedings, statements, and decisions made at a meeting.

[0174] "Action items" are specific tasks or action items that need to be carried out after the meeting ends.

[0175] "Robots" are automated devices that are placed in factories and perform real-time analysis and monitoring of meetings.

[0176] "Monitoring" refers to the act of watching the progress of a meeting and sending that data to a server or generating AI.

[0177] "Real-time analysis" refers to the process of analyzing data collected during a meeting on the spot and having the generating AI immediately take appropriate action.

[0178] This invention is a system that utilizes smart robots to streamline meetings held in factories. This system has the function of inputting the meeting schedule and content and using generative AI to assist the meeting.

[0179] System Configuration

[0180] Hardware:

[0181] Tablet or smartphone: A device that allows users to enter meeting schedules and details.

[0182] Server: Handles database and generation AI processing.

[0183] Smart robots: Deployed in factories, they monitor meetings and provide real-time analysis.

[0184] software:

[0185] Django or Flask: for server-side data processing and secure communication.

[0186] TensorFlow and PyTorch: For running and training generative AI models.

[0187] OpenCV: For robotic video and audio capture and analysis.

[0188] What the program does

[0189] 1. User Input:

[0190] Users input the meeting schedule (date, time, location) and content (agenda, participants) via a tablet or smartphone, as well as requests to the generation AI (questioners, supplementary explanations, etc.). This information is sent to the server via a dedicated web form.

[0191] 2. Prepare the generative AI:

[0192] The server analyzes the received meeting details and stores them in a database. Based on this information, it initializes a generative AI model and inputs the meeting content and roles. Here, TensorFlow or PyTorch are used to train and run the generative AI.

[0193] 3. Real-time meeting participation:

[0194] When a meeting begins, a smart robot located within the factory monitors the meeting. The robot uses libraries such as OpenCV to capture video and audio data and send it to a server. The server's generative AI analyzes this data in real time and asks appropriate questions or provides additional explanations.

[0195] 4. Automatic generation of meeting minutes:

[0196] After the meeting ends, the server automatically generates minutes based on the log data collected during the meeting. These minutes include the progress of the meeting, what was said, and what decisions were made. The generated minutes are automatically distributed to all participants.

[0197] 5. Extract action items:

[0198] The server extracts next action items from the generated minutes and organizes them into a list, which is also automatically distributed to all participants.

[0199] Specific examples

[0200] Below is an example of a meeting, along with the minutes and action items generated from it:

[0201] Prompt Sentence Examples

[0202] Meeting on new product marketing plan

[0203] Date and time: October 15, 2023 14:00

[0204] Location: Factory Conference Room B

[0205] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[0206] Agenda: New product launch plans, advertising strategies, sales forecasts

[0207] Generative AI role: Supplementary explanation

[0208] Example of generated meeting minutes and action items

[0209] minutes

[0210] Meeting Date and Time: October 15, 2023, 14:00

[0211] Location: Factory Conference Room B

[0212] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[0213] Agenda: New product launch plans, advertising strategies, sales forecasts

[0214] Statement:

[0215] Administrator A: The new product is scheduled to go on sale in November.

[0216] Administrator B: The advertising campaign is scheduled to begin in late October.

[0217] Manager C: Our sales forecast is for 10,000 units per year.

[0218] Action Items

[0219] Administrator A: Preparation for sales launch in November

[0220] Administrator B: Preparing for advertising campaign in late October

[0221] Manager C: Sales forecast specifics and schedule adjustments

[0222] As in this example, the generated minutes and action items allow for efficient follow-up of the meeting.

[0223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0224] Step 1:

[0225] Users use tablets or smartphones to enter meeting schedules and details. The entered data, including details such as date, time, location, agenda, and participants, is sent to the server via a web form. This information is then stored in a database on the server side.

[0226] Step 2:

[0227] The server analyzes the saved meeting details and prepares the generative AI. Specifically, it initializes the generative AI model (such as TensorFlow or PyTorch) based on the meeting content and roles, and loads the necessary libraries and models. The analysis results in meeting summary data.

[0228] Step 3:

[0229] When a meeting begins, smart robots deployed within the factory capture video and audio and send it to a server. This captured data is processed using libraries such as OpenCV and sent to the server as a live feed of the meeting in real time. The server then provides this input data to the generative AI.

[0230] Step 4:

[0231] The generation AI analyzes the live feed data in real time. On the server side, the generation AI generates questions and supplementary explanations appropriate for the progress of the meeting based on the input live feed data. This creates output messages according to the timing of speech.

[0232] Step 5:

[0233] During the meeting, questions and supplementary explanations posed by the generation AI are provided to meeting participants through the robot. The robot outputs audio and presents the generated messages during the meeting. This process helps the meeting discussion proceed smoothly.

[0234] Step 6:

[0235] Once the meeting is over, the server automatically generates minutes based on the log data collected during the meeting. These minutes are generated using Natural Language Processing (NLP) technology and output as a document containing the meeting history, statements made, decisions made, and more.

[0236] Step 7:

[0237] The server extracts next action items from the generated minutes and organizes them into a list. This list is also analyzed using NLP technology to generate an action item list that clearly indicates specific tasks and responsible persons.

[0238] Step 8:

[0239] Finally, the generated minutes and action item list are automatically distributed to all participants, primarily via email or an internal company sharing system, allowing participants to refer to the list and efficiently plan the next meeting or follow-up work.

[0240] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0241] The present invention is a system that combines generative AI and an emotion engine to improve the effectiveness of meetings, and detailed embodiments thereof will be described below.

[0242] Input of meeting schedule and content

[0243] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[0244] Preparing for generative AI and emotion engines

[0245] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[0246] Generative AI meeting participation

[0247] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[0248] Taking meeting minutes

[0249] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[0250] Extracting action items

[0251] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[0252] Specific examples

[0253] A concrete example is a meeting about a marketing plan for a new product.

[0254] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[0255] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion engine and initializes the user emotion recognition function.

[0256] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[0257] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[0258] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[0259] By combining generative AI with an emotion engine, the system of the present invention further improves the effectiveness of meetings, automates the creation of meeting minutes and the organization of action items, and simplifies post-meeting follow-up.

[0260] The processing flow will be explained below.

[0261] Step 1:

[0262] The user enters the meeting schedule and details from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[0263] Step 2:

[0264] The server receives and analyzes the data sent by the user, specifically checking information such as the meeting date, time, location, topic, agenda, and AI role, and converts it into a format that is easy to understand.

[0265] Step 3:

[0266] The server stores the formatted meeting details in a database, which manages details such as the meeting schedule, content, participant list, and the role of the generating AI.

[0267] Step 4:

[0268] The server initializes the generated AI, loading the necessary libraries and AI models, and allocating memory and CPU / GPU resources to prepare an instance of the generated AI.

[0269] Step 5:

[0270] The server inputs meeting details into the generation AI, specifically providing it with the meeting topic, set roles, and pre-set content.

[0271] Step 6:

[0272] The server initializes the emotion engine and prepares the emotion recognition module, which includes functions for recognizing the user's face, voice analysis, and text analysis.

[0273] Step 7:

[0274] The server streams the live feed (real-time data of the meeting) to the generation AI and emotion engine. When the meeting starts, real-time data such as audio, text, and video is provided to the generation AI and emotion engine.

[0275] Step 8:

[0276] The generative AI analyzes the live feed and asks appropriate questions as the meeting progresses, providing additional explanations based on emotions, etc. The emotion engine monitors changes in the user's emotions in real time and feeds that information back to the generative AI.

[0277] Step 9:

[0278] The server collects log data and emotion data during the meeting. It records and stores data on all statements, discussions, decisions, and important emotion changes during the meeting.

[0279] Step 10:

[0280] The server automatically generates minutes based on the collected log data and emotion data. The minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are then reviewed to check for errors or omissions.

[0281] Step 11:

[0282] The server distributes the generated minutes to participants after the meeting ends, via email or a dedicated system.

[0283] Step 12:

[0284] The server extracts next action items based on the meeting log data and emotion data. These action items list follow-up tasks and activities to be carried out before the next meeting.

[0285] Step 13:

[0286] The server distributes the generated action item list to participants after the meeting, improving follow-up efficiency and simplifying task management.

[0287] Example 2

[0288] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0289] In conventional conferencing systems, meeting content and progress management is often done manually, which tends to reduce meeting efficiency. It is also difficult to consider changes in participants' emotions during the meeting, which can lead to inappropriate questions or supplementary explanations not being given. Furthermore, creating meeting minutes and extracting action items takes time, which makes follow-up ineffective.

[0290] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0291] In this invention, the server includes a means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a storage device, a means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, and a means for preparing an emotion recognition engine and collecting user emotion data. This improves meeting efficiency and enables appropriate questions and supplementary explanations based on participants' emotions. Furthermore, the server can automatically generate minutes and extract and distribute next action items, enabling effective follow-up.

[0292] "Meeting schedule" refers to detailed information about a meeting, such as the date, time, location, participants, and agenda.

[0293] "Meeting content" refers to information including the topics or agenda to be discussed at the meeting, the procedure, and any necessary materials or information.

[0294] "Storage device" refers to a device or system for storing digital information, such as a database.

[0295] "Generative AI" refers to an artificial intelligence model that automatically generates questions and follow-up explanations as a meeting progresses.

[0296] "Emotion recognition engine" refers to a module that analyzes a user's voice, video, and text data to recognize their emotional state.

[0297] "Live Feed" means the stream of audio, video, and text data provided in real time during a conference.

[0298] "Log data" refers to data such as participants' comments, actions, and timestamps that are recorded during the meeting.

[0299] "Minutes" refers to an official record document that summarizes the progress of a meeting, what was said, and what decisions were made.

[0300] "Action items" refer to specific follow-up tasks or activities decided upon at the meeting.

[0301] This invention is a meeting support system that uses generative AI and an emotion recognition engine to improve the efficiency of meetings and simplify the automatic creation of minutes and management of follow-up tasks.

[0302] Input of meeting schedule and content

[0303] The user uses a device (PC or smartphone) to enter the meeting schedule and details into a dedicated input form. Specifically, the user fills in the form on the PC or smartphone with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, "questioner" or "supplementary explanation"), and clicks the send button.

[0304] Preparing for generative AI and emotion recognition engines

[0305] The server analyzes the conference information received from the user and stores it in a database. Specifically, it analyzes the received data in JSON format, extracts the necessary information, and records it in the database.

[0306] Next, the server prepares the generation AI. For example, the AI ​​model used here is OpenAI's GPT-4. The server provides the generation AI with a prompt based on the meeting content and initializes the generation AI.

[0307] The server also configures an emotion recognition engine. It installs facial recognition, voice analysis, and text analysis modules and configures them to recognize user emotions in real time. For example, we use the Microsoft Azure Emotion Analysis API.

[0308] Generative AI meeting participation

[0309] When the meeting begins, the server provides a live feed to the generation AI, including audio, video, and text data from the meeting. The generation AI analyzes the provided live feed and asks appropriate questions or provides supplementary explanations based on the user's emotional data. Changes in the user's emotions are monitored by an emotion recognition engine, and the data is also fed back to the generation AI in real time.

[0310] Taking meeting minutes

[0311] During the meeting, the server collects all log data, including audio, video, text, and emotion data. After the meeting ends, the server uses generative AI to create minutes based on the collected log data. The generated minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The minutes are distributed to all participants via email or chat tools.

[0312] Extracting action items

[0313] The server analyzes the meeting log data and emotion data to extract action items to be discussed at the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting. This streamlines the management of follow-up tasks decided in the meeting.

[0314] Specific examples

[0315] A concrete example is a meeting about a marketing plan for a new product.

[0316] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[0317] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion recognition engine and initializes the user emotion recognition function.

[0318] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[0319] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[0320] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[0321] Example prompts for generative AI models

[0322] "A user has scheduled a meeting to discuss a marketing plan for a new product. The meeting will start at 14:00 on October 15, 2023, and the generative AI will act as a supplementary explanation. Analyze the user's emotions in real time and ask appropriate questions or provide supplementary explanations based on the data from the emotion engine."

[0323] As described above, the system of the present invention can improve the efficiency of meetings and automate the creation of minutes and the management of action items.

[0324] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0325] Step 1:

[0326] The user inputs the conference information from the terminal.

[0327] The input details include the date and time (e.g., 15 October 2023, 2:00 PM), location (e.g., Conference Room A), topic (e.g., marketing plan for a new product), agenda (e.g., sharing market analysis results), and desired role of the generative AI (e.g., supplementary explanation). Specifically, users enter this information into the input form on their PC or smartphone and click the send button.

[0328] Input: Meeting date, time, location, topic, agenda, desired role of the AI

[0329] Output: Data sent from the device to the server

[0330] Step 2:

[0331] The device sends the input information to the server. The device uses an HTTP POST request to send the input data to the server's API endpoint.

[0332] Input: User-entered meeting information

[0333] Output: Meeting information data sent to the server

[0334] Step 3:

[0335] The server parses the received data. The server parses the meeting information in JSON format, extracts the necessary fields (date, time, topic, role, etc.), and stores them in the database. Specifically, it parses the data and stores the extracted information in the appropriate table.

[0336] Input: Meeting information data sent from the terminal

[0337] Output: Parsed meeting information, data stored in a database

[0338] Step 4:

[0339] The server initializes the generation AI, loads OpenAI's GPT-4 model, and provides the AI ​​with prompts based on the meeting content.

[0340] Input: Parsed meeting information (e.g., meeting date and time, new product marketing plan, additional explainers)

[0341] Output: Initialized generated AI

[0342] Step 5:

[0343] The server configures the emotion recognition engine. The emotion recognition engine installs facial recognition, voice analysis, and text analysis modules and is configured to recognize user emotions in real time. For example, the Microsoft Azure emotion analysis API is used.

[0344] Input: Server configuration information, emotion recognition engine

[0345] Output: Initialized emotion recognition engine

[0346] Step 6:

[0347] When a meeting starts, the server provides a live feed to the generation AI, capturing audio, video, and text data from the meeting in real time and sending it to the generation AI.

[0348] Input: Live meeting feed (audio, video, text)

[0349] Output: Live feed data provided to the generative AI

[0350] Step 7:

[0351] The generative AI analyzes the live feed and asks appropriate questions or provides additional explanations based on the user's emotional data. Emotional data from the emotion recognition engine is also collected and fed back to the generative AI.

[0352] Input: Live feed data, emotion data

[0353] Output: appropriate questions and further explanations

[0354] Step 8:

[0355] The server collects log data during the meeting, recording the meeting log in real time, including audio, video, text, and emotional data, and storing it in a database.

[0356] Input: Contents of remarks made during meetings, emotional data

[0357] Output: Collected and stored log data

[0358] Step 9:

[0359] After the meeting ends, minutes are created using a generation AI based on the log data collected by the server.

[0360] Input: Collected log data

[0361] Output: Generated meeting minutes

[0362] Step 10:

[0363] The server distributes the generated minutes to all participants via email or chat tools. Specifically, the minutes file is attached to an email and sent to the participants.

[0364] Input: Generated minutes

[0365] Output: Minutes distributed to participants

[0366] Step 11:

[0367] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[0368] Input: Meeting log data, emotion data

[0369] Output: Extracted action items list

[0370] Step 12:

[0371] The server distributes the list of action items to participants by attaching the list to an email with a message saying, "Please complete the following action items before the next meeting."

[0372] Input: Extracted action item list

[0373] Output: Action item list distributed to participants

[0374] (Application example 2)

[0375] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0376] Conventional conferencing systems provide insufficient support for improving meeting efficiency, and it is difficult to conduct meetings while taking into account the emotions of employees, especially in environments such as smart factories. Furthermore, creating meeting minutes and extracting action items requires a lot of time and effort.

[0377] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database; means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content; means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the usage content; means for collecting log data during the meeting, automatically generating minutes, and distributing them to participants; means for extracting next action items from the meeting log data and distributing them to participants; and means for analyzing employee emotions during the meeting in real time and feeding the emotion data back to the generation AI. This enables optimization of meeting progress through real-time emotion analysis and automatic generation of minutes and action items after the meeting ends.

[0378] A "meeting schedule" is planning information about the dates, times, locations, and topics of meetings.

[0379] "Meeting content" refers to information such as the agenda and topics to be discussed during the meeting, as well as planned remarks and materials.

[0380] A "database" is a system for storing and managing data in a structured way.

[0381] "Generative AI" is artificial intelligence that automatically generates text, questions, answers, etc. based on input information.

[0382] A "live feed" is a continuous stream of data, such as audio, video, or text, provided in real time.

[0383] "Emotional data" refers to information about an employee's emotions obtained through facial expressions and voice analysis.

[0384] Minutes are a document that records the statements and decisions made during a meeting.

[0385] "Action items" are follow-up tasks or next agenda items decided at the meeting.

[0386] This invention is a system that improves the effectiveness of meetings by combining a generative AI and an emotion engine. Specific embodiments for carrying out the invention will be described below.

[0387] 1. Input of meeting schedule and content

[0388] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[0389] 2. Preparing the generative AI and emotion engine

[0390] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[0391] In particular, the emotion engine uses OpenCV to detect faces from video feeds and analyze facial expressions, natural language processing libraries such as Spacy for speech analysis, and the generative AI uses the Transformers library from Hugging Face.

[0392] 3. Generative AI participation in meetings

[0393] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[0394] 4. Taking meeting minutes

[0395] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[0396] 5. Extracting action items

[0397] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[0398] Prompt Sentence Examples

[0399] "In today's meeting, we discussed the maintenance schedule and operational issues for a new product line. Please analyze what was said and extract important action items based on the following information."

[0400] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0401] Step 1:

[0402] Users input meeting schedules and content

[0403] Using a device (PC or smartphone), the user fills in the input form with the date, time, location, agenda, topic, and desired role of the generating AI (e.g., questioner or supplementary explanation), and clicks the send button. The input data is sent to the server.

[0404] Input: Meeting schedule information (date, time, location), meeting content (agenda, topic, role of the generation AI)

[0405] Output: Meeting schedule and contents saved on the server

[0406] Step 2:

[0407] The server analyzes the meeting information and stores it in a database.

[0408] The server parses the received meeting schedule and content data, extracts the necessary information, and stores it in a database, preparing the data for future reference and for use in initializing the generative AI.

[0409] Input: Meeting schedule and content data submitted by users

[0410] Output: Parsed meeting information stored in a database

[0411] Step 3:

[0412] The server prepares the generative AI and emotion engine

[0413] Based on the saved meeting information, the server initializes the generative AI, loads the necessary libraries and models (e.g., Hugging Face Transformers), prepares the emotion engine (e.g., OpenCV or Spacy), and installs and configures the facial recognition, speech analysis, and text analysis modules.

[0414] Input: Meeting information stored in the database

[0415] Output: Initialized generative AI, configured emotion engine

[0416] Step 4:

[0417] The server provides a live feed to the generation AI, feeding back emotional data.

[0418] When a meeting begins, the server provides a live feed of the meeting (audio, video, and text data) to the Generator AI, which analyzes the data and asks appropriate questions or provides additional explanations. The Emotion Engine analyzes employees' emotions in real time and provides feedback on the emotion data to the Generator AI.

[0419] Input: Live feed (audio, video, text data), emotion data

[0420] Output: Appropriate questions and supplementary explanations from generative AI

[0421] Step 5:

[0422] The server collects meeting log data and automatically generates minutes.

[0423] The server collects audio, video, and text data from the meeting as logs, and then, using pre-set rules and the power of generative AI, automatically creates minutes of the meeting and distributes them to all participants after the meeting has ended.

[0424] Input: Audio, video, and text data during the meeting

[0425] Output: Generated meeting minutes

[0426] Step 6:

[0427] The server extracts the action items and distributes them to the participants.

[0428] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[0429] Input: Meeting log data, emotion data

[0430] Output: Action item list

[0431] By dividing the specific processing flow into steps in this way, we aim to make the "Form for carrying out the invention" as a patent specification more clearly understandable.

[0432] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0433] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0434] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0435] [Second embodiment]

[0436] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0437] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0438] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0439] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0440] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0441] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0442] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0443] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0444] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0445] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0446] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0447] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0448] The present invention is a system for improving the effectiveness of meetings by utilizing generative AI, and detailed embodiments thereof will be described below.

[0449] Input of meeting schedule and content

[0450] Users input the meeting schedule and details into the server from their devices (PCs or smartphones). Specifically, they fill out an input form with detailed information such as the meeting date, time, location, topic, agenda, and the role they want the AI ​​to play (e.g., questioner, supplementary explanation), and then click the send button.

[0451] Preparing the Generative AI

[0452] The server analyzes the received meeting details and stores them in a database. It prepares the generative AI based on the stored information and loads the necessary libraries and models to input the meeting content and roles into the AI.

[0453] Generative AI meeting participation

[0454] The server provides a live feed (real-time data of the meeting) to the AI ​​generator. Once the meeting begins, the AI ​​generator analyzes the comments and discussions in real time, asking appropriate questions and providing supplementary explanations.

[0455] Taking meeting minutes

[0456] The server collects log data during the meeting and automatically generates minutes, which include the date and time of the meeting, participants, agenda, comments made, and decisions made. These minutes are distributed to participants after the meeting ends.

[0457] Extracting action items

[0458] The server extracts next action items from the meeting log data. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[0459] Specific examples

[0460] A concrete example is a meeting about a marketing plan for a new product.

[0461] 1. The user enters the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[0462] 2. The server receives this information, initializes the generating AI, and inputs the meeting contents into the AI.

[0463] 3. Once the meeting begins, the generative AI analyzes the live meeting feed and provides additional explanations at the appropriate time.

[0464] 4. The server automatically generates minutes from the log data during the meeting and distributes them to participants after the meeting ends.

[0465] 5. The server extracts next action items from the meeting log data, organizes them into a list, and distributes it to participants.

[0466] In this way, the system of the present invention increases the effectiveness of meetings and simplifies post-meeting follow-up by automating the creation of meeting minutes and organizing action items.

[0467] The processing flow will be explained below.

[0468] Step 1:

[0469] The user enters the meeting schedule and content from a device (PC or smartphone), filling in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and then clicks the submit button.

[0470] Step 2:

[0471] The server parses the incoming data, checking what the user has entered and converting it to the required format, checking for consistency of data format and the presence of all required fields.

[0472] Step 3:

[0473] The server stores the formatted meeting details in a database, including the meeting schedule, content, participant list, and the role of the generating AI.

[0474] Step 4:

[0475] The server initializes the generated AI, loading the necessary libraries and AI models, and preparing the generated AI instance. Memory and CPU / GPU resources are also allocated at this stage.

[0476] Step 5:

[0477] The server inputs detailed information about the meeting into the AI ​​generator, including the meeting topic and assigned roles, and instructs the AI ​​on the next task to be performed.

[0478] Step 6:

[0479] The server streams the live feed (real-time data of the meeting) to the generation AI. After the meeting starts, it provides the generation AI with data such as audio, text, and video that is generated in real time.

[0480] Step 7:

[0481] The generative AI analyzes what is being said in real time during the meeting. As the meeting progresses, the generative AI monitors what is being said and discussed in real time, asking appropriate questions and providing supplementary explanations as needed.

[0482] Step 8:

[0483] The server collects log data during the meeting, recording all statements, discussions, and decisions made in the meeting as logs and saving them for later analysis.

[0484] Step 9:

[0485] The server automatically generates minutes based on the meeting log data. These minutes include the meeting date and time, participants, agenda, comments, and decisions. The generated minutes are then checked for errors and omissions.

[0486] Step 10:

[0487] The server distributes the generated minutes to participants after the meeting ends. The minutes are sent to all participants via email or a dedicated system.

[0488] Step 11:

[0489] The server extracts the next action items from the meeting log data, analyzes the log data, and lists the tasks and follow-up items that need to be done by the next meeting.

[0490] Step 12:

[0491] The server distributes the generated action item list to participants, and the list is sent to all participants via email or a dedicated system, improving follow-up efficiency.

[0492] Example 1

[0493] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0494] Conventional meeting support systems have problems with insufficient understanding of meeting content and efficient follow-up, resulting in time and effort being required even after the meeting has ended. Furthermore, it is difficult to provide appropriate supplementary explanations and questions in real time based on the content and progress of comments made during the meeting, which means that the effectiveness of the meeting cannot be maximized. Furthermore, follow-up tasks such as creating minutes and organizing action items are often done manually, and there is a need for greater efficiency.

[0495] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0496] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the content of the meeting, means for collecting log data during the meeting, automatically generating minutes and distributing them to participants, means for extracting next action items from the meeting log data and distributing them to participants, means for acquiring real-time data from the conference tool and providing it to the generation AI, means for the generation AI to analyze remarks in real time and generate questions and provide supplementary explanations, and means for distributing the automatically generated minutes and action item list to participants after the meeting, thereby enabling efficient meetings and effective follow-up.

[0497] The "input meeting schedule and meeting content" is data that the user inputs into the server via the terminal, including detailed information such as the start date and time of the meeting, the location, the topic, and the agenda.

[0498] A "database" is a storage device for efficiently storing and managing detailed information about analyzed meetings, log data, and configuration information for the generation AI.

[0499] "Generative AI" is an artificial intelligence model that asks appropriate questions and provides supplementary explanations based on the content of a meeting, and is a system that operates using specific libraries and algorithms.

[0500] "Live feed" refers to the audio and text data generated in real time during a meeting, which the server provides to the generating AI, allowing the generating AI to participate in the meeting.

[0501] "Log data" refers to a record of all statements made during a meeting and other related data that is later used to create minutes and action items.

[0502] Minutes are documents that summarize the main points of a meeting by organizing what was said, decisions made, and agenda items, and are distributed to participants after the meeting has ended.

[0503] "Action items" are tasks and follow-up matters decided during the meeting that must be completed before the next meeting. By extracting and distributing these to participants, it becomes clear what needs to be completed before the next meeting.

[0504] "Real-time data" refers to the audio and text information generated in real time during a meeting, which is then analyzed by the generating AI.

[0505] The present invention is a system that uses generative AI to improve the efficiency and usefulness of meetings. Specific embodiments for carrying out the present invention will be described in detail below.

[0506] Hardware and Software Use

[0507] This system uses a server, a terminal, and a generative AI model. The terminal is a personal computer or smartphone that communicates with the server. The server has powerful computing power and runs the generative AI model. The following specific software is used:

[0508] Database management systems: MySQL or PostgreSQL

[0509] AI model libraries: TensorFlow, PyTorch

[0510] Communication protocol: HTTPS

[0511] Natural language processing technology: Natural language analysis libraries (e.g., spaCy, NLTK)

[0512] Meeting tool: General online meeting tools (e.g., Zoom or Microsoft Teams)

[0513] System Operation

[0514] The user uses the terminal to input the schedule and content of the meeting. For example, in the case of a meeting about a marketing plan for a new product, the user would input the following:

[0515] Meeting Schedule: October 15, 2023, 14:00

[0516] Meeting topic: New product marketing plan

[0517] The role of generative AI: Supplementary explanation

[0518] When the user clicks the submit button, the input data is sent to the server.

[0519] The server analyzes the received meeting information, converts the meeting details into JSON format, and stores it in a database. Next, it loads the libraries and models required to prepare the generative AI model (e.g., a generative AI model using TensorFlow) and configures it according to the content and roles of the meeting.

[0520] When a meeting starts, the server acquires real-time data from the meeting tool. This data includes audio and text information generated during the meeting. The real-time data is provided to the generation AI, which analyzes what is being said. For example, the generation AI generates appropriate questions and supplementary explanations in real time and outputs them as text through the meeting tool.

[0521] The server also collects all log data during the meeting. After the meeting, minutes are automatically generated based on the log data. The generated minutes include the date and time, participants, agenda, comments, and decisions. The minutes are then distributed to participants via email.

[0522] Furthermore, the server extracts action items from the log data, including next follow-up tasks, and distributes them to participants as a list, allowing for efficient management of tasks decided in the meeting.

[0523] In this way, the invention centralizes meeting efficiency and follow-up management, and by using generative AI, it is possible to maximize the usefulness of meetings.

[0524] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0525] Step 1:

[0526] The user inputs the schedule and details of the meeting from the terminal and sends it.

[0527] A user launches a conference management application using a PC or smartphone.

[0528] In the input form, enter the date, time, location, topic, agenda, and desired role of the generating AI for the meeting.

[0529] Click the submit button to send the input data to the server.

[0530] (Input) Meeting schedule, content, and generation AI role

[0531] (Output) Meeting information sent to the server

[0532] Step 2:

[0533] The server analyzes the received conference information and stores it in a database.

[0534] The server converts the received conference information into JSON format.

[0535] Save the JSON format data to the database.

[0536] (Input) Meeting information received from the user

[0537] (Output) Meeting information stored in the database

[0538] Step 3:

[0539] The server prepares the generative AI model.

[0540] The meeting information is obtained from the database and the generating AI is initialized according to the content and role of the meeting.

[0541] Loading a generative AI model (e.g. using TensorFlow) and required libraries.

[0542] (Input) Meeting information stored in the database

[0543] (Output) Initialized generated AI

[0544] Step 4:

[0545] The server collects real-time data from the meeting and provides it to the generation AI.

[0546] When a meeting starts, the server obtains real-time audio and text data from the meeting tool (e.g., Zoom, Microsoft Teams).

[0547] The acquired real-time data is passed to the generation AI.

[0548] (Input) Real-time data from conferencing tools

[0549] (Output) Real-time data provided to the generative AI

[0550] Step 5:

[0551] The generative AI analyzes the content of statements in real time and generates appropriate questions and supplementary explanations.

[0552] The generative AI analyzes real-time data and generates necessary questions and supplementary explanations based on what is being said and the progress of the meeting.

[0553] The generated questions and supplementary explanations are output to the conference tool and displayed to participants in real time.

[0554] (Input) Real-time data and meeting information

[0555] (Output) Generated questions and supplementary explanations

[0556] Step 6:

[0557] The server collects all log data during the meeting and automatically generates minutes.

[0558] Accumulates log data obtained from conference tools.

[0559] Minutes are automatically generated from the accumulated log data using a minutes generation algorithm.

[0560] The generated minutes include the date, time, participants, agenda, what was said, and what was decided.

[0561] (Input) Log data during the meeting

[0562] (Output) Automatically generated minutes

[0563] Step 7:

[0564] The server distributes the minutes and action item list to participants.

[0565] Automatically generated meeting minutes and a list of action items extracted from the log data.

[0566] The generated minutes and action item list are distributed to participants via email or other means.

[0567] (Input) Automatically generated minutes and action items

[0568] (Output) Meeting minutes and action item list distributed to participants

[0569] The above steps will improve the efficiency and usefulness of meetings.

[0570] (Application example 1)

[0571] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0572] In modern manufacturing, the effectiveness of meetings and their follow-up are important for maximizing factory operational efficiency. However, the traditional manual process of creating meeting minutes and extracting action items requires a great deal of time and effort, resulting in a decline in meeting efficiency. Furthermore, it can be difficult to ask questions or provide supplementary explanations in real time, which can sometimes prevent meetings from proceeding smoothly. To solve this problem, a more efficient meeting support system is needed.

[0573] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0574] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the utilization content, and means for a robot located in the factory to monitor the meeting and analyze the progress of the meeting in real time to provide assistance. This increases the effectiveness of meetings and enables the automatic generation of minutes and the extraction and distribution of action items.

[0575] "Meeting schedule" refers to information that includes the date, time, location and associated timeline of a meeting.

[0576] "Meeting content" refers to information including the agenda, topics, and participant roles in a meeting.

[0577] A "database" is an information management system that stores conference details and allows them to be retrieved at a later time.

[0578] "Generative AI" is a system that uses artificial intelligence technology to automatically generate questions and supplementary explanations based on the content of the meeting.

[0579] "Live feed" refers to streaming data that sends real-time information about the progress of a meeting to the generation AI.

[0580] "Questions" are posed by the generative AI in response to comments made or topics discussed during a meeting, and are intended to deepen the discussion.

[0581] "Supplementary explanation" refers to additional information or explanations that make the content of the meeting easier to understand.

[0582] "Log data" refers to digital data that records what was said during a meeting and the progress of the discussion.

[0583] "Minutes" are official documents summarizing the proceedings, statements, and decisions made at a meeting.

[0584] "Action items" are specific tasks or action items that need to be carried out after the meeting ends.

[0585] "Robots" are automated devices that are placed in factories and perform real-time analysis and monitoring of meetings.

[0586] "Monitoring" refers to the act of watching the progress of a meeting and sending that data to a server or generating AI.

[0587] "Real-time analysis" refers to the process of analyzing data collected during a meeting on the spot and having the generating AI immediately take appropriate action.

[0588] This invention is a system that utilizes smart robots to streamline meetings held in factories. This system has the function of inputting the meeting schedule and content and using generative AI to assist the meeting.

[0589] System Configuration

[0590] Hardware:

[0591] Tablet or smartphone: A device that allows users to enter meeting schedules and details.

[0592] Server: Handles database and generation AI processing.

[0593] Smart robots: Deployed in factories, they monitor meetings and provide real-time analysis.

[0594] software:

[0595] Django or Flask: for server-side data processing and secure communication.

[0596] TensorFlow and PyTorch: For running and training generative AI models.

[0597] OpenCV: For robotic video and audio capture and analysis.

[0598] What the program does

[0599] 1. User Input:

[0600] Users input the meeting schedule (date, time, location) and content (agenda, participants) via a tablet or smartphone, as well as requests to the generation AI (questioners, supplementary explanations, etc.). This information is sent to the server via a dedicated web form.

[0601] 2. Prepare the generative AI:

[0602] The server analyzes the received meeting details and stores them in a database. Based on this information, it initializes a generative AI model and inputs the meeting content and roles. Here, TensorFlow or PyTorch are used to train and run the generative AI.

[0603] 3. Real-time meeting participation:

[0604] When a meeting begins, a smart robot located within the factory monitors the meeting. The robot uses libraries such as OpenCV to capture video and audio data and send it to a server. The server's generative AI analyzes this data in real time and asks appropriate questions or provides additional explanations.

[0605] 4. Automatic generation of meeting minutes:

[0606] After the meeting ends, the server automatically generates minutes based on the log data collected during the meeting. These minutes include the progress of the meeting, what was said, and what decisions were made. The generated minutes are automatically distributed to all participants.

[0607] 5. Extract action items:

[0608] The server extracts next action items from the generated minutes and organizes them into a list, which is also automatically distributed to all participants.

[0609] Specific examples

[0610] Below is an example of a meeting, along with the minutes and action items generated from it:

[0611] Prompt Sentence Examples

[0612] Meeting on new product marketing plan

[0613] Date and time: October 15, 2023 14:00

[0614] Location: Factory Conference Room B

[0615] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[0616] Agenda: New product launch plans, advertising strategies, sales forecasts

[0617] Generative AI role: Supplementary explanation

[0618] Example of generated meeting minutes and action items

[0619] minutes

[0620] Meeting Date and Time: October 15, 2023, 14:00

[0621] Location: Factory Conference Room B

[0622] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[0623] Agenda: New product launch plans, advertising strategies, sales forecasts

[0624] Statement:

[0625] Administrator A: The new product is scheduled to go on sale in November.

[0626] Administrator B: The advertising campaign is scheduled to begin in late October.

[0627] Manager C: Our sales forecast is for 10,000 units per year.

[0628] Action Items

[0629] Administrator A: Preparation for sales launch in November

[0630] Administrator B: Preparing for advertising campaign in late October

[0631] Manager C: Sales forecast specifics and schedule adjustments

[0632] As in this example, the generated minutes and action items allow for efficient follow-up of the meeting.

[0633] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0634] Step 1:

[0635] Users use tablets or smartphones to enter meeting schedules and details. The entered data, including details such as date, time, location, agenda, and participants, is sent to the server via a web form. This information is then stored in a database on the server side.

[0636] Step 2:

[0637] The server analyzes the saved meeting details and prepares the generative AI. Specifically, it initializes the generative AI model (such as TensorFlow or PyTorch) based on the meeting content and roles, and loads the necessary libraries and models. The analysis results in meeting summary data.

[0638] Step 3:

[0639] When a meeting begins, smart robots deployed within the factory capture video and audio and send it to a server. This captured data is processed using libraries such as OpenCV and sent to the server as a live feed of the meeting in real time. The server then provides this input data to the generative AI.

[0640] Step 4:

[0641] The generation AI analyzes the live feed data in real time. On the server side, the generation AI generates questions and supplementary explanations appropriate for the progress of the meeting based on the input live feed data. This creates output messages according to the timing of speech.

[0642] Step 5:

[0643] During the meeting, questions and supplementary explanations posed by the generation AI are provided to meeting participants through the robot. The robot outputs audio and presents the generated messages during the meeting. This process helps the meeting discussion proceed smoothly.

[0644] Step 6:

[0645] Once the meeting is over, the server automatically generates minutes based on the log data collected during the meeting. These minutes are generated using Natural Language Processing (NLP) technology and output as a document containing the meeting history, statements made, decisions made, and more.

[0646] Step 7:

[0647] The server extracts next action items from the generated minutes and organizes them into a list. This list is also analyzed using NLP technology to generate an action item list that clearly indicates specific tasks and responsible persons.

[0648] Step 8:

[0649] Finally, the generated minutes and action item list are automatically distributed to all participants, primarily via email or an internal company sharing system, allowing participants to refer to the list and efficiently plan the next meeting or follow-up work.

[0650] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0651] The present invention is a system that combines generative AI and an emotion engine to improve the effectiveness of meetings, and detailed embodiments thereof will be described below.

[0652] Input of meeting schedule and content

[0653] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[0654] Preparing for generative AI and emotion engines

[0655] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[0656] Generative AI meeting participation

[0657] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[0658] Taking meeting minutes

[0659] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[0660] Extracting action items

[0661] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[0662] Specific examples

[0663] A concrete example is a meeting about a marketing plan for a new product.

[0664] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[0665] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion engine and initializes the user emotion recognition function.

[0666] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[0667] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[0668] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[0669] By combining generative AI with an emotion engine, the system of the present invention further improves the effectiveness of meetings, automates the creation of meeting minutes and the organization of action items, and simplifies post-meeting follow-up.

[0670] The processing flow will be explained below.

[0671] Step 1:

[0672] The user enters the meeting schedule and details from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[0673] Step 2:

[0674] The server receives and analyzes the data sent by the user, specifically checking information such as the meeting date, time, location, topic, agenda, and AI role, and converts it into a format that is easy to understand.

[0675] Step 3:

[0676] The server stores the formatted meeting details in a database, which manages details such as the meeting schedule, content, participant list, and the role of the generating AI.

[0677] Step 4:

[0678] The server initializes the generated AI, loading the necessary libraries and AI models, and allocating memory and CPU / GPU resources to prepare an instance of the generated AI.

[0679] Step 5:

[0680] The server inputs meeting details into the generation AI, specifically providing it with the meeting topic, set roles, and pre-set content.

[0681] Step 6:

[0682] The server initializes the emotion engine and prepares the emotion recognition module, which includes functions for recognizing the user's face, voice analysis, and text analysis.

[0683] Step 7:

[0684] The server streams the live feed (real-time data of the meeting) to the generation AI and emotion engine. When the meeting starts, real-time data such as audio, text, and video is provided to the generation AI and emotion engine.

[0685] Step 8:

[0686] The generative AI analyzes the live feed and asks appropriate questions as the meeting progresses, providing additional explanations based on emotions, etc. The emotion engine monitors changes in the user's emotions in real time and feeds that information back to the generative AI.

[0687] Step 9:

[0688] The server collects log data and emotion data during the meeting. It records and stores data on all statements, discussions, decisions, and important emotion changes during the meeting.

[0689] Step 10:

[0690] The server automatically generates minutes based on the collected log data and emotion data. The minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are then reviewed to check for errors or omissions.

[0691] Step 11:

[0692] The server distributes the generated minutes to participants after the meeting ends, via email or a dedicated system.

[0693] Step 12:

[0694] The server extracts next action items based on the meeting log data and emotion data. These action items list follow-up tasks and activities to be carried out before the next meeting.

[0695] Step 13:

[0696] The server distributes the generated action item list to participants after the meeting, improving follow-up efficiency and simplifying task management.

[0697] Example 2

[0698] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] In conventional conferencing systems, meeting content and progress management is often done manually, which tends to reduce meeting efficiency. It is also difficult to consider changes in participants' emotions during the meeting, which can lead to inappropriate questions or supplementary explanations not being given. Furthermore, creating meeting minutes and extracting action items takes time, which makes follow-up ineffective.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0701] In this invention, the server includes a means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a storage device, a means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, and a means for preparing an emotion recognition engine and collecting user emotion data. This improves meeting efficiency and enables appropriate questions and supplementary explanations based on participants' emotions. Furthermore, the server can automatically generate minutes and extract and distribute next action items, enabling effective follow-up.

[0702] "Meeting schedule" refers to detailed information about a meeting, such as the date, time, location, participants, and agenda.

[0703] "Meeting content" refers to information including the topics or agenda to be discussed at the meeting, the procedure, and any necessary materials or information.

[0704] "Storage device" refers to a device or system for storing digital information, such as a database.

[0705] "Generative AI" refers to an artificial intelligence model that automatically generates questions and follow-up explanations as a meeting progresses.

[0706] "Emotion recognition engine" refers to a module that analyzes a user's voice, video, and text data to recognize their emotional state.

[0707] "Live Feed" means the stream of audio, video, and text data provided in real time during a conference.

[0708] "Log data" refers to data such as participants' comments, actions, and timestamps that are recorded during the meeting.

[0709] "Minutes" refers to an official record document that summarizes the progress of a meeting, what was said, and what decisions were made.

[0710] "Action items" refer to specific follow-up tasks or activities decided upon at the meeting.

[0711] This invention is a meeting support system that uses generative AI and an emotion recognition engine to improve the efficiency of meetings and simplify the automatic creation of minutes and management of follow-up tasks.

[0712] Input of meeting schedule and content

[0713] The user uses a device (PC or smartphone) to enter the meeting schedule and details into a dedicated input form. Specifically, the user fills in the form on the PC or smartphone with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, "questioner" or "supplementary explanation"), and clicks the send button.

[0714] Preparing for generative AI and emotion recognition engines

[0715] The server analyzes the conference information received from the user and stores it in a database. Specifically, it analyzes the received data in JSON format, extracts the necessary information, and records it in the database.

[0716] Next, the server prepares the generation AI. For example, the AI ​​model used here is OpenAI's GPT-4. The server provides the generation AI with a prompt based on the meeting content and initializes the generation AI.

[0717] The server also configures an emotion recognition engine. It installs facial recognition, voice analysis, and text analysis modules and configures them to recognize user emotions in real time. For example, we use the Microsoft Azure Emotion Analysis API.

[0718] Generative AI meeting participation

[0719] When the meeting begins, the server provides a live feed to the generation AI, including audio, video, and text data from the meeting. The generation AI analyzes the provided live feed and asks appropriate questions or provides supplementary explanations based on the user's emotional data. Changes in the user's emotions are monitored by an emotion recognition engine, and the data is also fed back to the generation AI in real time.

[0720] Taking meeting minutes

[0721] During the meeting, the server collects all log data, including audio, video, text, and emotion data. After the meeting ends, the server uses generative AI to create minutes based on the collected log data. The generated minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The minutes are distributed to all participants via email or chat tools.

[0722] Extracting action items

[0723] The server analyzes the meeting log data and emotion data to extract action items to be discussed at the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting. This streamlines the management of follow-up tasks decided in the meeting.

[0724] Specific examples

[0725] A concrete example is a meeting about a marketing plan for a new product.

[0726] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[0727] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion recognition engine and initializes the user emotion recognition function.

[0728] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[0729] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[0730] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[0731] Example prompts for generative AI models

[0732] "A user has scheduled a meeting to discuss a marketing plan for a new product. The meeting will start at 14:00 on October 15, 2023, and the generative AI will act as a supplementary explanation. Analyze the user's emotions in real time and ask appropriate questions or provide supplementary explanations based on the data from the emotion engine."

[0733] As described above, the system of the present invention can improve the efficiency of meetings and automate the creation of minutes and the management of action items.

[0734] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0735] Step 1:

[0736] The user inputs the conference information from the terminal.

[0737] The input details include the date and time (e.g., 15 October 2023, 2:00 PM), location (e.g., Conference Room A), topic (e.g., marketing plan for a new product), agenda (e.g., sharing market analysis results), and desired role of the generative AI (e.g., supplementary explanation). Specifically, users enter this information into the input form on their PC or smartphone and click the send button.

[0738] Input: Meeting date, time, location, topic, agenda, desired role of the AI

[0739] Output: Data sent from the device to the server

[0740] Step 2:

[0741] The device sends the input information to the server. The device uses an HTTP POST request to send the input data to the server's API endpoint.

[0742] Input: User-entered meeting information

[0743] Output: Meeting information data sent to the server

[0744] Step 3:

[0745] The server parses the received data. The server parses the meeting information in JSON format, extracts the necessary fields (date, time, topic, role, etc.), and stores them in the database. Specifically, it parses the data and stores the extracted information in the appropriate table.

[0746] Input: Meeting information data sent from the terminal

[0747] Output: Parsed meeting information, data stored in a database

[0748] Step 4:

[0749] The server initializes the generation AI, loads OpenAI's GPT-4 model, and provides the AI ​​with prompts based on the meeting content.

[0750] Input: Parsed meeting information (e.g., meeting date and time, new product marketing plan, additional explainers)

[0751] Output: Initialized generated AI

[0752] Step 5:

[0753] The server configures the emotion recognition engine. The emotion recognition engine installs facial recognition, voice analysis, and text analysis modules and is configured to recognize user emotions in real time. For example, the Microsoft Azure emotion analysis API is used.

[0754] Input: Server configuration information, emotion recognition engine

[0755] Output: Initialized emotion recognition engine

[0756] Step 6:

[0757] When a meeting starts, the server provides a live feed to the generation AI, capturing audio, video, and text data from the meeting in real time and sending it to the generation AI.

[0758] Input: Live meeting feed (audio, video, text)

[0759] Output: Live feed data provided to the generative AI

[0760] Step 7:

[0761] The generative AI analyzes the live feed and asks appropriate questions or provides additional explanations based on the user's emotional data. Emotional data from the emotion recognition engine is also collected and fed back to the generative AI.

[0762] Input: Live feed data, emotion data

[0763] Output: appropriate questions and further explanations

[0764] Step 8:

[0765] The server collects log data during the meeting, recording the meeting log in real time, including audio, video, text, and emotional data, and storing it in a database.

[0766] Input: Contents of remarks made during meetings, emotional data

[0767] Output: Collected and stored log data

[0768] Step 9:

[0769] After the meeting ends, minutes are created using a generation AI based on the log data collected by the server.

[0770] Input: Collected log data

[0771] Output: Generated meeting minutes

[0772] Step 10:

[0773] The server distributes the generated minutes to all participants via email or chat tools. Specifically, the minutes file is attached to an email and sent to the participants.

[0774] Input: Generated minutes

[0775] Output: Minutes distributed to participants

[0776] Step 11:

[0777] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[0778] Input: Meeting log data, emotion data

[0779] Output: Extracted action items list

[0780] Step 12:

[0781] The server distributes the list of action items to participants by attaching the list to an email with a message saying, "Please complete the following action items before the next meeting."

[0782] Input: Extracted action item list

[0783] Output: Action item list distributed to participants

[0784] (Application example 2)

[0785] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0786] Conventional conferencing systems provide insufficient support for improving meeting efficiency, and it is difficult to conduct meetings while taking into account the emotions of employees, especially in environments such as smart factories. Furthermore, creating meeting minutes and extracting action items requires a lot of time and effort.

[0787] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database; means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content; means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the usage content; means for collecting log data during the meeting, automatically generating minutes, and distributing them to participants; means for extracting next action items from the meeting log data and distributing them to participants; and means for analyzing employee emotions during the meeting in real time and feeding the emotion data back to the generation AI. This enables optimization of meeting progress through real-time emotion analysis and automatic generation of minutes and action items after the meeting ends.

[0788] A "meeting schedule" is planning information about the dates, times, locations, and topics of meetings.

[0789] "Meeting content" refers to information such as the agenda and topics to be discussed during the meeting, as well as planned remarks and materials.

[0790] A "database" is a system for storing and managing data in a structured way.

[0791] "Generative AI" is artificial intelligence that automatically generates text, questions, answers, etc. based on input information.

[0792] A "live feed" is a continuous stream of data, such as audio, video, or text, provided in real time.

[0793] "Emotional data" refers to information about an employee's emotions obtained through facial expressions and voice analysis.

[0794] Minutes are a document that records the statements and decisions made during a meeting.

[0795] "Action items" are follow-up tasks or next agenda items decided at the meeting.

[0796] This invention is a system that improves the effectiveness of meetings by combining a generative AI and an emotion engine. Specific embodiments for carrying out the invention will be described below.

[0797] 1. Input of meeting schedule and content

[0798] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[0799] 2. Preparing the generative AI and emotion engine

[0800] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[0801] In particular, the emotion engine uses OpenCV to detect faces from video feeds and analyze facial expressions, natural language processing libraries such as Spacy for speech analysis, and the generative AI uses the Transformers library from Hugging Face.

[0802] 3. Generative AI participation in meetings

[0803] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[0804] 4. Taking meeting minutes

[0805] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[0806] 5. Extracting action items

[0807] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[0808] Prompt Sentence Examples

[0809] "In today's meeting, we discussed the maintenance schedule and operational issues for a new product line. Please analyze what was said and extract important action items based on the following information."

[0810] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0811] Step 1:

[0812] Users input meeting schedules and content

[0813] Using a device (PC or smartphone), the user fills in the input form with the date, time, location, agenda, topic, and desired role of the generating AI (e.g., questioner or supplementary explanation), and clicks the send button. The input data is sent to the server.

[0814] Input: Meeting schedule information (date, time, location), meeting content (agenda, topic, role of the generation AI)

[0815] Output: Meeting schedule and contents saved on the server

[0816] Step 2:

[0817] The server analyzes the meeting information and stores it in a database.

[0818] The server parses the received meeting schedule and content data, extracts the necessary information, and stores it in a database, preparing the data for future reference and for use in initializing the generative AI.

[0819] Input: Meeting schedule and content data submitted by users

[0820] Output: Parsed meeting information stored in a database

[0821] Step 3:

[0822] The server prepares the generative AI and emotion engine

[0823] Based on the saved meeting information, the server initializes the generative AI, loads the necessary libraries and models (e.g., Hugging Face Transformers), prepares the emotion engine (e.g., OpenCV or Spacy), and installs and configures the facial recognition, speech analysis, and text analysis modules.

[0824] Input: Meeting information stored in the database

[0825] Output: Initialized generative AI, configured emotion engine

[0826] Step 4:

[0827] The server provides a live feed to the generation AI, feeding back emotional data.

[0828] When a meeting begins, the server provides a live feed of the meeting (audio, video, and text data) to the Generator AI, which analyzes the data and asks appropriate questions or provides additional explanations. The Emotion Engine analyzes employees' emotions in real time and provides feedback on the emotion data to the Generator AI.

[0829] Input: Live feed (audio, video, text data), emotion data

[0830] Output: Appropriate questions and supplementary explanations from generative AI

[0831] Step 5:

[0832] The server collects meeting log data and automatically generates minutes.

[0833] The server collects audio, video, and text data from the meeting as logs, and then, using pre-set rules and the power of generative AI, automatically creates minutes of the meeting and distributes them to all participants after the meeting has ended.

[0834] Input: Audio, video, and text data during the meeting

[0835] Output: Generated meeting minutes

[0836] Step 6:

[0837] The server extracts the action items and distributes them to the participants.

[0838] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[0839] Input: Meeting log data, emotion data

[0840] Output: Action item list

[0841] By dividing the specific processing flow into steps in this way, we aim to make the "Form for carrying out the invention" as a patent specification more clearly understandable.

[0842] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0843] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0844] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0845] [Third embodiment]

[0846] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0847] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0848] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0849] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0850] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0851] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0852] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0853] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0854] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0855] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0856] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0857] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0858] The present invention is a system for improving the effectiveness of meetings by utilizing generative AI, and detailed embodiments thereof will be described below.

[0859] Input of meeting schedule and content

[0860] Users input the meeting schedule and details into the server from their devices (PCs or smartphones). Specifically, they fill out an input form with detailed information such as the meeting date, time, location, topic, agenda, and the role they want the AI ​​to play (e.g., questioner, supplementary explanation), and then click the send button.

[0861] Preparing the Generative AI

[0862] The server analyzes the received meeting details and stores them in a database. It prepares the generative AI based on the stored information and loads the necessary libraries and models to input the meeting content and roles into the AI.

[0863] Generative AI meeting participation

[0864] The server provides a live feed (real-time data of the meeting) to the AI ​​generator. Once the meeting begins, the AI ​​generator analyzes the comments and discussions in real time, asking appropriate questions and providing supplementary explanations.

[0865] Taking meeting minutes

[0866] The server collects log data during the meeting and automatically generates minutes, which include the date and time of the meeting, participants, agenda, comments made, and decisions made. These minutes are distributed to participants after the meeting ends.

[0867] Extracting action items

[0868] The server extracts next action items from the meeting log data. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[0869] Specific examples

[0870] A concrete example is a meeting about a marketing plan for a new product.

[0871] 1. The user enters the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[0872] 2. The server receives this information, initializes the generating AI, and inputs the meeting contents into the AI.

[0873] 3. Once the meeting begins, the generative AI analyzes the live meeting feed and provides additional explanations at the appropriate time.

[0874] 4. The server automatically generates minutes from the log data during the meeting and distributes them to participants after the meeting ends.

[0875] 5. The server extracts next action items from the meeting log data, organizes them into a list, and distributes it to participants.

[0876] In this way, the system of the present invention increases the effectiveness of meetings and simplifies post-meeting follow-up by automating the creation of meeting minutes and organizing action items.

[0877] The processing flow will be explained below.

[0878] Step 1:

[0879] The user enters the meeting schedule and content from a device (PC or smartphone), filling in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and then clicks the submit button.

[0880] Step 2:

[0881] The server parses the incoming data, checking what the user has entered and converting it to the required format, checking for consistency of data format and the presence of all required fields.

[0882] Step 3:

[0883] The server stores the formatted meeting details in a database, including the meeting schedule, content, participant list, and the role of the generating AI.

[0884] Step 4:

[0885] The server initializes the generated AI, loading the necessary libraries and AI models, and preparing the generated AI instance. Memory and CPU / GPU resources are also allocated at this stage.

[0886] Step 5:

[0887] The server inputs detailed information about the meeting into the AI ​​generator, including the meeting topic and assigned roles, and instructs the AI ​​on the next task to be performed.

[0888] Step 6:

[0889] The server streams the live feed (real-time data of the meeting) to the generation AI. After the meeting starts, it provides the generation AI with data such as audio, text, and video that is generated in real time.

[0890] Step 7:

[0891] The generative AI analyzes what is being said in real time during the meeting. As the meeting progresses, the generative AI monitors what is being said and discussed in real time, asking appropriate questions and providing supplementary explanations as needed.

[0892] Step 8:

[0893] The server collects log data during the meeting, recording all statements, discussions, and decisions made in the meeting as logs and saving them for later analysis.

[0894] Step 9:

[0895] The server automatically generates minutes based on the meeting log data. These minutes include the meeting date and time, participants, agenda, comments, and decisions. The generated minutes are then checked for errors and omissions.

[0896] Step 10:

[0897] The server distributes the generated minutes to participants after the meeting ends. The minutes are sent to all participants via email or a dedicated system.

[0898] Step 11:

[0899] The server extracts the next action items from the meeting log data, analyzes the log data, and lists the tasks and follow-up items that need to be done by the next meeting.

[0900] Step 12:

[0901] The server distributes the generated action item list to participants, and the list is sent to all participants via email or a dedicated system, improving follow-up efficiency.

[0902] Example 1

[0903] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0904] Conventional meeting support systems have problems with insufficient understanding of meeting content and efficient follow-up, resulting in time and effort being required even after the meeting has ended. Furthermore, it is difficult to provide appropriate supplementary explanations and questions in real time based on the content and progress of comments made during the meeting, which means that the effectiveness of the meeting cannot be maximized. Furthermore, follow-up tasks such as creating minutes and organizing action items are often done manually, and there is a need for greater efficiency.

[0905] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0906] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the content of the meeting, means for collecting log data during the meeting, automatically generating minutes and distributing them to participants, means for extracting next action items from the meeting log data and distributing them to participants, means for acquiring real-time data from the conference tool and providing it to the generation AI, means for the generation AI to analyze remarks in real time and generate questions and provide supplementary explanations, and means for distributing the automatically generated minutes and action item list to participants after the meeting, thereby enabling efficient meetings and effective follow-up.

[0907] The "input meeting schedule and meeting content" is data that the user inputs into the server via the terminal, including detailed information such as the start date and time of the meeting, the location, the topic, and the agenda.

[0908] A "database" is a storage device for efficiently storing and managing detailed information about analyzed meetings, log data, and configuration information for the generation AI.

[0909] "Generative AI" is an artificial intelligence model that asks appropriate questions and provides supplementary explanations based on the content of a meeting, and is a system that operates using specific libraries and algorithms.

[0910] "Live feed" refers to the audio and text data generated in real time during a meeting, which the server provides to the generating AI, allowing the generating AI to participate in the meeting.

[0911] "Log data" refers to a record of all statements made during a meeting and other related data that is later used to create minutes and action items.

[0912] Minutes are documents that summarize the main points of a meeting by organizing what was said, decisions made, and agenda items, and are distributed to participants after the meeting has ended.

[0913] "Action items" are tasks and follow-up matters decided during the meeting that must be completed before the next meeting. By extracting and distributing these to participants, it becomes clear what needs to be completed before the next meeting.

[0914] "Real-time data" refers to the audio and text information generated in real time during a meeting, which is then analyzed by the generating AI.

[0915] The present invention is a system that uses generative AI to improve the efficiency and usefulness of meetings. Specific embodiments for carrying out the present invention will be described in detail below.

[0916] Hardware and Software Use

[0917] This system uses a server, a terminal, and a generative AI model. The terminal is a personal computer or smartphone that communicates with the server. The server has powerful computing power and runs the generative AI model. The following specific software is used:

[0918] Database management systems: MySQL or PostgreSQL

[0919] AI model libraries: TensorFlow, PyTorch

[0920] Communication protocol: HTTPS

[0921] Natural language processing technology: Natural language analysis libraries (e.g., spaCy, NLTK)

[0922] Meeting tool: General online meeting tools (e.g., Zoom or Microsoft Teams)

[0923] System Operation

[0924] The user uses the terminal to input the schedule and content of the meeting. For example, in the case of a meeting about a marketing plan for a new product, the user would input the following:

[0925] Meeting Schedule: October 15, 2023, 14:00

[0926] Meeting topic: New product marketing plan

[0927] The role of generative AI: Supplementary explanation

[0928] When the user clicks the submit button, the input data is sent to the server.

[0929] The server analyzes the received meeting information, converts the meeting details into JSON format, and stores it in a database. Next, it loads the libraries and models required to prepare the generative AI model (e.g., a generative AI model using TensorFlow) and configures it according to the content and roles of the meeting.

[0930] When a meeting starts, the server acquires real-time data from the meeting tool. This data includes audio and text information generated during the meeting. The real-time data is provided to the generation AI, which analyzes what is being said. For example, the generation AI generates appropriate questions and supplementary explanations in real time and outputs them as text through the meeting tool.

[0931] The server also collects all log data during the meeting. After the meeting, minutes are automatically generated based on the log data. The generated minutes include the date and time, participants, agenda, comments, and decisions. The minutes are then distributed to participants via email.

[0932] Furthermore, the server extracts action items from the log data, including next follow-up tasks, and distributes them to participants as a list, allowing for efficient management of tasks decided in the meeting.

[0933] In this way, the invention centralizes meeting efficiency and follow-up management, and by using generative AI, it is possible to maximize the usefulness of meetings.

[0934] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0935] Step 1:

[0936] The user inputs the schedule and details of the meeting from the terminal and sends it.

[0937] A user launches a conference management application using a PC or smartphone.

[0938] In the input form, enter the date, time, location, topic, agenda, and desired role of the generating AI for the meeting.

[0939] Click the submit button to send the input data to the server.

[0940] (Input) Meeting schedule, content, and generation AI role

[0941] (Output) Meeting information sent to the server

[0942] Step 2:

[0943] The server analyzes the received conference information and stores it in a database.

[0944] The server converts the received conference information into JSON format.

[0945] Save the JSON format data to the database.

[0946] (Input) Meeting information received from the user

[0947] (Output) Meeting information stored in the database

[0948] Step 3:

[0949] The server prepares the generative AI model.

[0950] The meeting information is obtained from the database and the generating AI is initialized according to the content and role of the meeting.

[0951] Loading a generative AI model (e.g. using TensorFlow) and required libraries.

[0952] (Input) Meeting information stored in the database

[0953] (Output) Initialized generated AI

[0954] Step 4:

[0955] The server collects real-time data from the meeting and provides it to the generation AI.

[0956] When a meeting starts, the server obtains real-time audio and text data from the meeting tool (e.g., Zoom, Microsoft Teams).

[0957] The acquired real-time data is passed to the generation AI.

[0958] (Input) Real-time data from conferencing tools

[0959] (Output) Real-time data provided to the generative AI

[0960] Step 5:

[0961] The generative AI analyzes the content of statements in real time and generates appropriate questions and supplementary explanations.

[0962] The generative AI analyzes real-time data and generates necessary questions and supplementary explanations based on what is being said and the progress of the meeting.

[0963] The generated questions and supplementary explanations are output to the conference tool and displayed to participants in real time.

[0964] (Input) Real-time data and meeting information

[0965] (Output) Generated questions and supplementary explanations

[0966] Step 6:

[0967] The server collects all log data during the meeting and automatically generates minutes.

[0968] Accumulates log data obtained from conference tools.

[0969] Minutes are automatically generated from the accumulated log data using a minutes generation algorithm.

[0970] The generated minutes include the date, time, participants, agenda, what was said, and what was decided.

[0971] (Input) Log data during the meeting

[0972] (Output) Automatically generated minutes

[0973] Step 7:

[0974] The server distributes the minutes and action item list to participants.

[0975] Automatically generated meeting minutes and a list of action items extracted from the log data.

[0976] The generated minutes and action item list are distributed to participants via email or other means.

[0977] (Input) Automatically generated minutes and action items

[0978] (Output) Meeting minutes and action item list distributed to participants

[0979] The above steps will improve the efficiency and usefulness of meetings.

[0980] (Application example 1)

[0981] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0982] In modern manufacturing, the effectiveness of meetings and their follow-up are important for maximizing factory operational efficiency. However, the traditional manual process of creating meeting minutes and extracting action items requires a great deal of time and effort, resulting in a decline in meeting efficiency. Furthermore, it can be difficult to ask questions or provide supplementary explanations in real time, which can sometimes prevent meetings from proceeding smoothly. To solve this problem, a more efficient meeting support system is needed.

[0983] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0984] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the utilization content, and means for a robot located in the factory to monitor the meeting and analyze the progress of the meeting in real time to provide assistance. This increases the effectiveness of meetings and enables the automatic generation of minutes and the extraction and distribution of action items.

[0985] "Meeting schedule" refers to information that includes the date, time, location and associated timeline of a meeting.

[0986] "Meeting content" refers to information including the agenda, topics, and participant roles in a meeting.

[0987] A "database" is an information management system that stores conference details and allows them to be retrieved at a later time.

[0988] "Generative AI" is a system that uses artificial intelligence technology to automatically generate questions and supplementary explanations based on the content of the meeting.

[0989] "Live feed" refers to streaming data that sends real-time information about the progress of a meeting to the generation AI.

[0990] "Questions" are posed by the generative AI in response to comments made or topics discussed during a meeting, and are intended to deepen the discussion.

[0991] "Supplementary explanation" refers to additional information or explanations that make the content of the meeting easier to understand.

[0992] "Log data" refers to digital data that records what was said during a meeting and the progress of the discussion.

[0993] "Minutes" are official documents summarizing the proceedings, statements, and decisions made at a meeting.

[0994] "Action items" are specific tasks or action items that need to be carried out after the meeting ends.

[0995] "Robots" are automated devices that are placed in factories and perform real-time analysis and monitoring of meetings.

[0996] "Monitoring" refers to the act of watching the progress of a meeting and sending that data to a server or generating AI.

[0997] "Real-time analysis" refers to the process of analyzing data collected during a meeting on the spot and having the generating AI immediately take appropriate action.

[0998] This invention is a system that utilizes smart robots to streamline meetings held in factories. This system has the function of inputting the meeting schedule and content and using generative AI to assist the meeting.

[0999] System Configuration

[1000] Hardware:

[1001] Tablet or smartphone: A device that allows users to enter meeting schedules and details.

[1002] Server: Handles database and generation AI processing.

[1003] Smart robots: Deployed in factories, they monitor meetings and provide real-time analysis.

[1004] software:

[1005] Django or Flask: for server-side data processing and secure communication.

[1006] TensorFlow and PyTorch: For running and training generative AI models.

[1007] OpenCV: For robotic video and audio capture and analysis.

[1008] What the program does

[1009] 1. User Input:

[1010] Users input the meeting schedule (date, time, location) and content (agenda, participants) via a tablet or smartphone, as well as requests to the generation AI (questioners, supplementary explanations, etc.). This information is sent to the server via a dedicated web form.

[1011] 2. Prepare the generative AI:

[1012] The server analyzes the received meeting details and stores them in a database. Based on this information, it initializes a generative AI model and inputs the meeting content and roles. Here, TensorFlow or PyTorch are used to train and run the generative AI.

[1013] 3. Real-time meeting participation:

[1014] When a meeting begins, a smart robot located within the factory monitors the meeting. The robot uses libraries such as OpenCV to capture video and audio data and send it to a server. The server's generative AI analyzes this data in real time and asks appropriate questions or provides additional explanations.

[1015] 4. Automatic generation of meeting minutes:

[1016] After the meeting ends, the server automatically generates minutes based on the log data collected during the meeting. These minutes include the progress of the meeting, what was said, and what decisions were made. The generated minutes are automatically distributed to all participants.

[1017] 5. Extract action items:

[1018] The server extracts next action items from the generated minutes and organizes them into a list, which is also automatically distributed to all participants.

[1019] Specific examples

[1020] Below is an example of a meeting, along with the minutes and action items generated from it:

[1021] Prompt Sentence Examples

[1022] Meeting on new product marketing plan

[1023] Date and time: October 15, 2023 14:00

[1024] Location: Factory Conference Room B

[1025] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[1026] Agenda: New product launch plans, advertising strategies, sales forecasts

[1027] Generative AI role: Supplementary explanation

[1028] Example of generated meeting minutes and action items

[1029] minutes

[1030] Meeting Date and Time: October 15, 2023, 14:00

[1031] Location: Factory Conference Room B

[1032] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[1033] Agenda: New product launch plans, advertising strategies, sales forecasts

[1034] Statement:

[1035] Administrator A: The new product is scheduled to go on sale in November.

[1036] Administrator B: The advertising campaign is scheduled to begin in late October.

[1037] Manager C: Our sales forecast is for 10,000 units per year.

[1038] Action Items

[1039] Administrator A: Preparation for sales launch in November

[1040] Administrator B: Preparing for advertising campaign in late October

[1041] Manager C: Sales forecast specifics and schedule adjustments

[1042] As in this example, the generated minutes and action items allow for efficient follow-up of the meeting.

[1043] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1044] Step 1:

[1045] Users use tablets or smartphones to enter meeting schedules and details. The entered data, including details such as date, time, location, agenda, and participants, is sent to the server via a web form. This information is then stored in a database on the server side.

[1046] Step 2:

[1047] The server analyzes the saved meeting details and prepares the generative AI. Specifically, it initializes the generative AI model (such as TensorFlow or PyTorch) based on the meeting content and roles, and loads the necessary libraries and models. The analysis results in meeting summary data.

[1048] Step 3:

[1049] When a meeting begins, smart robots deployed within the factory capture video and audio and send it to a server. This captured data is processed using libraries such as OpenCV and sent to the server as a live feed of the meeting in real time. The server then provides this input data to the generative AI.

[1050] Step 4:

[1051] The generation AI analyzes the live feed data in real time. On the server side, the generation AI generates questions and supplementary explanations appropriate for the progress of the meeting based on the input live feed data. This creates output messages according to the timing of speech.

[1052] Step 5:

[1053] During the meeting, questions and supplementary explanations posed by the generation AI are provided to meeting participants through the robot. The robot outputs audio and presents the generated messages during the meeting. This process helps the meeting discussion proceed smoothly.

[1054] Step 6:

[1055] Once the meeting is over, the server automatically generates minutes based on the log data collected during the meeting. These minutes are generated using Natural Language Processing (NLP) technology and output as a document containing the meeting history, statements made, decisions made, and more.

[1056] Step 7:

[1057] The server extracts next action items from the generated minutes and organizes them into a list. This list is also analyzed using NLP technology to generate an action item list that clearly indicates specific tasks and responsible persons.

[1058] Step 8:

[1059] Finally, the generated minutes and action item list are automatically distributed to all participants, primarily via email or an internal company sharing system, allowing participants to refer to the list and efficiently plan the next meeting or follow-up work.

[1060] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1061] The present invention is a system that combines generative AI and an emotion engine to improve the effectiveness of meetings, and detailed embodiments thereof will be described below.

[1062] Input of meeting schedule and content

[1063] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[1064] Preparing for generative AI and emotion engines

[1065] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[1066] Generative AI meeting participation

[1067] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[1068] Taking meeting minutes

[1069] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[1070] Extracting action items

[1071] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[1072] Specific examples

[1073] A concrete example is a meeting about a marketing plan for a new product.

[1074] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[1075] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion engine and initializes the user emotion recognition function.

[1076] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[1077] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[1078] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[1079] By combining generative AI with an emotion engine, the system of the present invention further improves the effectiveness of meetings, automates the creation of meeting minutes and the organization of action items, and simplifies post-meeting follow-up.

[1080] The processing flow will be explained below.

[1081] Step 1:

[1082] The user enters the meeting schedule and details from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[1083] Step 2:

[1084] The server receives and analyzes the data sent by the user, specifically checking information such as the meeting date, time, location, topic, agenda, and AI role, and converts it into a format that is easy to understand.

[1085] Step 3:

[1086] The server stores the formatted meeting details in a database, which manages details such as the meeting schedule, content, participant list, and the role of the generating AI.

[1087] Step 4:

[1088] The server initializes the generated AI, loading the necessary libraries and AI models, and allocating memory and CPU / GPU resources to prepare an instance of the generated AI.

[1089] Step 5:

[1090] The server inputs meeting details into the generation AI, specifically providing it with the meeting topic, set roles, and pre-set content.

[1091] Step 6:

[1092] The server initializes the emotion engine and prepares the emotion recognition module, which includes functions for recognizing the user's face, voice analysis, and text analysis.

[1093] Step 7:

[1094] The server streams the live feed (real-time data of the meeting) to the generation AI and emotion engine. When the meeting starts, real-time data such as audio, text, and video is provided to the generation AI and emotion engine.

[1095] Step 8:

[1096] The generative AI analyzes the live feed and asks appropriate questions as the meeting progresses, providing additional explanations based on emotions, etc. The emotion engine monitors changes in the user's emotions in real time and feeds that information back to the generative AI.

[1097] Step 9:

[1098] The server collects log data and emotion data during the meeting. It records and stores data on all statements, discussions, decisions, and important emotion changes during the meeting.

[1099] Step 10:

[1100] The server automatically generates minutes based on the collected log data and emotion data. The minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are then reviewed to check for errors or omissions.

[1101] Step 11:

[1102] The server distributes the generated minutes to participants after the meeting ends, via email or a dedicated system.

[1103] Step 12:

[1104] The server extracts next action items based on the meeting log data and emotion data. These action items list follow-up tasks and activities to be carried out before the next meeting.

[1105] Step 13:

[1106] The server distributes the generated action item list to participants after the meeting, improving follow-up efficiency and simplifying task management.

[1107] Example 2

[1108] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1109] In conventional conferencing systems, meeting content and progress management is often done manually, which tends to reduce meeting efficiency. It is also difficult to consider changes in participants' emotions during the meeting, which can lead to inappropriate questions or supplementary explanations not being given. Furthermore, creating meeting minutes and extracting action items takes time, which makes follow-up ineffective.

[1110] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1111] In this invention, the server includes a means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a storage device, a means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, and a means for preparing an emotion recognition engine and collecting user emotion data. This improves meeting efficiency and enables appropriate questions and supplementary explanations based on participants' emotions. Furthermore, the server can automatically generate minutes and extract and distribute next action items, enabling effective follow-up.

[1112] "Meeting schedule" refers to detailed information about a meeting, such as the date, time, location, participants, and agenda.

[1113] "Meeting content" refers to information including the topics or agenda to be discussed at the meeting, the procedure, and any necessary materials or information.

[1114] "Storage device" refers to a device or system for storing digital information, such as a database.

[1115] "Generative AI" refers to an artificial intelligence model that automatically generates questions and follow-up explanations as a meeting progresses.

[1116] "Emotion recognition engine" refers to a module that analyzes a user's voice, video, and text data to recognize their emotional state.

[1117] "Live Feed" means the stream of audio, video, and text data provided in real time during a conference.

[1118] "Log data" refers to data such as participants' comments, actions, and timestamps that are recorded during the meeting.

[1119] "Minutes" refers to an official record document that summarizes the progress of a meeting, what was said, and what decisions were made.

[1120] "Action items" refer to specific follow-up tasks or activities decided upon at the meeting.

[1121] This invention is a meeting support system that uses generative AI and an emotion recognition engine to improve the efficiency of meetings and simplify the automatic creation of minutes and management of follow-up tasks.

[1122] Input of meeting schedule and content

[1123] The user uses a device (PC or smartphone) to enter the meeting schedule and details into a dedicated input form. Specifically, the user fills in the form on the PC or smartphone with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, "questioner" or "supplementary explanation"), and clicks the send button.

[1124] Preparing for generative AI and emotion recognition engines

[1125] The server analyzes the conference information received from the user and stores it in a database. Specifically, it analyzes the received data in JSON format, extracts the necessary information, and records it in the database.

[1126] Next, the server prepares the generation AI. For example, the AI ​​model used here is OpenAI's GPT-4. The server provides the generation AI with a prompt based on the meeting content and initializes the generation AI.

[1127] The server also configures an emotion recognition engine. It installs facial recognition, voice analysis, and text analysis modules and configures them to recognize user emotions in real time. For example, we use the Microsoft Azure Emotion Analysis API.

[1128] Generative AI meeting participation

[1129] When the meeting begins, the server provides a live feed to the generation AI, including audio, video, and text data from the meeting. The generation AI analyzes the provided live feed and asks appropriate questions or provides supplementary explanations based on the user's emotional data. Changes in the user's emotions are monitored by an emotion recognition engine, and the data is also fed back to the generation AI in real time.

[1130] Taking meeting minutes

[1131] During the meeting, the server collects all log data, including audio, video, text, and emotion data. After the meeting ends, the server uses generative AI to create minutes based on the collected log data. The generated minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The minutes are distributed to all participants via email or chat tools.

[1132] Extracting action items

[1133] The server analyzes the meeting log data and emotion data to extract action items to be discussed at the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting. This streamlines the management of follow-up tasks decided in the meeting.

[1134] Specific examples

[1135] A concrete example is a meeting about a marketing plan for a new product.

[1136] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[1137] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion recognition engine and initializes the user emotion recognition function.

[1138] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[1139] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[1140] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[1141] Example prompts for generative AI models

[1142] "A user has scheduled a meeting to discuss a marketing plan for a new product. The meeting will start at 14:00 on October 15, 2023, and the generative AI will act as a supplementary explanation. Analyze the user's emotions in real time and ask appropriate questions or provide supplementary explanations based on the data from the emotion engine."

[1143] As described above, the system of the present invention can improve the efficiency of meetings and automate the creation of minutes and the management of action items.

[1144] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1145] Step 1:

[1146] The user inputs the conference information from the terminal.

[1147] The input details include the date and time (e.g., 15 October 2023, 2:00 PM), location (e.g., Conference Room A), topic (e.g., marketing plan for a new product), agenda (e.g., sharing market analysis results), and desired role of the generative AI (e.g., supplementary explanation). Specifically, users enter this information into the input form on their PC or smartphone and click the send button.

[1148] Input: Meeting date, time, location, topic, agenda, desired role of the AI

[1149] Output: Data sent from the device to the server

[1150] Step 2:

[1151] The device sends the input information to the server. The device uses an HTTP POST request to send the input data to the server's API endpoint.

[1152] Input: User-entered meeting information

[1153] Output: Meeting information data sent to the server

[1154] Step 3:

[1155] The server parses the received data. The server parses the meeting information in JSON format, extracts the necessary fields (date, time, topic, role, etc.), and stores them in the database. Specifically, it parses the data and stores the extracted information in the appropriate table.

[1156] Input: Meeting information data sent from the terminal

[1157] Output: Parsed meeting information, data stored in a database

[1158] Step 4:

[1159] The server initializes the generation AI, loads OpenAI's GPT-4 model, and provides the AI ​​with prompts based on the meeting content.

[1160] Input: Parsed meeting information (e.g., meeting date and time, new product marketing plan, additional explainers)

[1161] Output: Initialized generated AI

[1162] Step 5:

[1163] The server configures the emotion recognition engine. The emotion recognition engine installs facial recognition, voice analysis, and text analysis modules and is configured to recognize user emotions in real time. For example, the Microsoft Azure emotion analysis API is used.

[1164] Input: Server configuration information, emotion recognition engine

[1165] Output: Initialized emotion recognition engine

[1166] Step 6:

[1167] When a meeting starts, the server provides a live feed to the generation AI, capturing audio, video, and text data from the meeting in real time and sending it to the generation AI.

[1168] Input: Live meeting feed (audio, video, text)

[1169] Output: Live feed data provided to the generative AI

[1170] Step 7:

[1171] The generative AI analyzes the live feed and asks appropriate questions or provides additional explanations based on the user's emotional data. Emotional data from the emotion recognition engine is also collected and fed back to the generative AI.

[1172] Input: Live feed data, emotion data

[1173] Output: appropriate questions and further explanations

[1174] Step 8:

[1175] The server collects log data during the meeting, recording the meeting log in real time, including audio, video, text, and emotional data, and storing it in a database.

[1176] Input: Contents of remarks made during meetings, emotional data

[1177] Output: Collected and stored log data

[1178] Step 9:

[1179] After the meeting ends, minutes are created using a generation AI based on the log data collected by the server.

[1180] Input: Collected log data

[1181] Output: Generated meeting minutes

[1182] Step 10:

[1183] The server distributes the generated minutes to all participants via email or chat tools. Specifically, the minutes file is attached to an email and sent to the participants.

[1184] Input: Generated minutes

[1185] Output: Minutes distributed to participants

[1186] Step 11:

[1187] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[1188] Input: Meeting log data, emotion data

[1189] Output: Extracted action items list

[1190] Step 12:

[1191] The server distributes the list of action items to participants by attaching the list to an email with a message saying, "Please complete the following action items before the next meeting."

[1192] Input: Extracted action item list

[1193] Output: Action item list distributed to participants

[1194] (Application example 2)

[1195] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1196] Conventional conferencing systems provide insufficient support for improving meeting efficiency, and it is difficult to conduct meetings while taking into account the emotions of employees, especially in environments such as smart factories. Furthermore, creating meeting minutes and extracting action items requires a lot of time and effort.

[1197] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database; means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content; means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the usage content; means for collecting log data during the meeting, automatically generating minutes, and distributing them to participants; means for extracting next action items from the meeting log data and distributing them to participants; and means for analyzing employee emotions during the meeting in real time and feeding the emotion data back to the generation AI. This enables optimization of meeting progress through real-time emotion analysis and automatic generation of minutes and action items after the meeting ends.

[1198] A "meeting schedule" is planning information about the dates, times, locations, and topics of meetings.

[1199] "Meeting content" refers to information such as the agenda and topics to be discussed during the meeting, as well as planned remarks and materials.

[1200] A "database" is a system for storing and managing data in a structured way.

[1201] "Generative AI" is artificial intelligence that automatically generates text, questions, answers, etc. based on input information.

[1202] A "live feed" is a continuous stream of data, such as audio, video, or text, provided in real time.

[1203] "Emotional data" refers to information about an employee's emotions obtained through facial expressions and voice analysis.

[1204] Minutes are a document that records the statements and decisions made during a meeting.

[1205] "Action items" are follow-up tasks or next agenda items decided at the meeting.

[1206] This invention is a system that improves the effectiveness of meetings by combining a generative AI and an emotion engine. Specific embodiments for carrying out the invention will be described below.

[1207] 1. Input of meeting schedule and content

[1208] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[1209] 2. Preparing the generative AI and emotion engine

[1210] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[1211] In particular, the emotion engine uses OpenCV to detect faces from video feeds and analyze facial expressions, natural language processing libraries such as Spacy for speech analysis, and the generative AI uses the Transformers library from Hugging Face.

[1212] 3. Generative AI participation in meetings

[1213] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[1214] 4. Taking meeting minutes

[1215] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[1216] 5. Extracting action items

[1217] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[1218] Prompt Sentence Examples

[1219] "In today's meeting, we discussed the maintenance schedule and operational issues for a new product line. Please analyze what was said and extract important action items based on the following information."

[1220] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1221] Step 1:

[1222] Users input meeting schedules and content

[1223] Using a device (PC or smartphone), the user fills in the input form with the date, time, location, agenda, topic, and desired role of the generating AI (e.g., questioner or supplementary explanation), and clicks the send button. The input data is sent to the server.

[1224] Input: Meeting schedule information (date, time, location), meeting content (agenda, topic, role of the generation AI)

[1225] Output: Meeting schedule and contents saved on the server

[1226] Step 2:

[1227] The server analyzes the meeting information and stores it in a database.

[1228] The server parses the received meeting schedule and content data, extracts the necessary information, and stores it in a database, preparing the data for future reference and for use in initializing the generative AI.

[1229] Input: Meeting schedule and content data submitted by users

[1230] Output: Parsed meeting information stored in a database

[1231] Step 3:

[1232] The server prepares the generative AI and emotion engine

[1233] Based on the saved meeting information, the server initializes the generative AI, loads the necessary libraries and models (e.g., Hugging Face Transformers), prepares the emotion engine (e.g., OpenCV or Spacy), and installs and configures the facial recognition, speech analysis, and text analysis modules.

[1234] Input: Meeting information stored in the database

[1235] Output: Initialized generative AI, configured emotion engine

[1236] Step 4:

[1237] The server provides a live feed to the generation AI, feeding back emotional data.

[1238] When a meeting begins, the server provides a live feed of the meeting (audio, video, and text data) to the Generator AI, which analyzes the data and asks appropriate questions or provides additional explanations. The Emotion Engine analyzes employees' emotions in real time and provides feedback on the emotion data to the Generator AI.

[1239] Input: Live feed (audio, video, text data), emotion data

[1240] Output: Appropriate questions and supplementary explanations from generative AI

[1241] Step 5:

[1242] The server collects meeting log data and automatically generates minutes.

[1243] The server collects audio, video, and text data from the meeting as logs, and then, using pre-set rules and the power of generative AI, automatically creates minutes of the meeting and distributes them to all participants after the meeting has ended.

[1244] Input: Audio, video, and text data during the meeting

[1245] Output: Generated meeting minutes

[1246] Step 6:

[1247] The server extracts the action items and distributes them to the participants.

[1248] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[1249] Input: Meeting log data, emotion data

[1250] Output: Action item list

[1251] By dividing the specific processing flow into steps in this way, we aim to make the "Form for carrying out the invention" as a patent specification more clearly understandable.

[1252] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1253] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1254] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1255] [Fourth embodiment]

[1256] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1257] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1258] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1259] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1260] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1261] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1262] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1263] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1264] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1265] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1266] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1267] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1268] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1269] The present invention is a system for improving the effectiveness of meetings by utilizing generative AI, and detailed embodiments thereof will be described below.

[1270] Input of meeting schedule and content

[1271] Users input the meeting schedule and details into the server from their devices (PCs or smartphones). Specifically, they fill out an input form with detailed information such as the meeting date, time, location, topic, agenda, and the role they want the AI ​​to play (e.g., questioner, supplementary explanation), and then click the send button.

[1272] Preparing the Generative AI

[1273] The server analyzes the received meeting details and stores them in a database. It prepares the generative AI based on the stored information and loads the necessary libraries and models to input the meeting content and roles into the AI.

[1274] Generative AI meeting participation

[1275] The server provides a live feed (real-time data of the meeting) to the AI ​​generator. Once the meeting begins, the AI ​​generator analyzes the comments and discussions in real time, asking appropriate questions and providing supplementary explanations.

[1276] Taking meeting minutes

[1277] The server collects log data during the meeting and automatically generates minutes, which include the date and time of the meeting, participants, agenda, comments made, and decisions made. These minutes are distributed to participants after the meeting ends.

[1278] Extracting action items

[1279] The server extracts next action items from the meeting log data. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[1280] Specific examples

[1281] A concrete example is a meeting about a marketing plan for a new product.

[1282] 1. The user enters the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[1283] 2. The server receives this information, initializes the generating AI, and inputs the meeting contents into the AI.

[1284] 3. Once the meeting begins, the generative AI analyzes the live meeting feed and provides additional explanations at the appropriate time.

[1285] 4. The server automatically generates minutes from the log data during the meeting and distributes them to participants after the meeting ends.

[1286] 5. The server extracts next action items from the meeting log data, organizes them into a list, and distributes it to participants.

[1287] In this way, the system of the present invention increases the effectiveness of meetings and simplifies post-meeting follow-up by automating the creation of meeting minutes and organizing action items.

[1288] The processing flow will be explained below.

[1289] Step 1:

[1290] The user enters the meeting schedule and content from a device (PC or smartphone), filling in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and then clicks the submit button.

[1291] Step 2:

[1292] The server parses the incoming data, checking what the user has entered and converting it to the required format, checking for consistency of data format and the presence of all required fields.

[1293] Step 3:

[1294] The server stores the formatted meeting details in a database, including the meeting schedule, content, participant list, and the role of the generating AI.

[1295] Step 4:

[1296] The server initializes the generated AI, loading the necessary libraries and AI models, and preparing the generated AI instance. Memory and CPU / GPU resources are also allocated at this stage.

[1297] Step 5:

[1298] The server inputs detailed information about the meeting into the AI ​​generator, including the meeting topic and assigned roles, and instructs the AI ​​on the next task to be performed.

[1299] Step 6:

[1300] The server streams the live feed (real-time data of the meeting) to the generation AI. After the meeting starts, it provides the generation AI with data such as audio, text, and video that is generated in real time.

[1301] Step 7:

[1302] The generative AI analyzes what is being said in real time during the meeting. As the meeting progresses, the generative AI monitors what is being said and discussed in real time, asking appropriate questions and providing supplementary explanations as needed.

[1303] Step 8:

[1304] The server collects log data during the meeting, recording all statements, discussions, and decisions made in the meeting as logs and saving them for later analysis.

[1305] Step 9:

[1306] The server automatically generates minutes based on the meeting log data. These minutes include the meeting date and time, participants, agenda, comments, and decisions. The generated minutes are then checked for errors and omissions.

[1307] Step 10:

[1308] The server distributes the generated minutes to participants after the meeting ends. The minutes are sent to all participants via email or a dedicated system.

[1309] Step 11:

[1310] The server extracts the next action items from the meeting log data, analyzes the log data, and lists the tasks and follow-up items that need to be done by the next meeting.

[1311] Step 12:

[1312] The server distributes the generated action item list to participants, and the list is sent to all participants via email or a dedicated system, improving follow-up efficiency.

[1313] Example 1

[1314] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1315] Conventional meeting support systems have problems with insufficient understanding of meeting content and efficient follow-up, resulting in time and effort being required even after the meeting has ended. Furthermore, it is difficult to provide appropriate supplementary explanations and questions in real time based on the content and progress of comments made during the meeting, which means that the effectiveness of the meeting cannot be maximized. Furthermore, follow-up tasks such as creating minutes and organizing action items are often done manually, and there is a need for greater efficiency.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1317] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the content of the meeting, means for collecting log data during the meeting, automatically generating minutes and distributing them to participants, means for extracting next action items from the meeting log data and distributing them to participants, means for acquiring real-time data from the conference tool and providing it to the generation AI, means for the generation AI to analyze remarks in real time and generate questions and provide supplementary explanations, and means for distributing the automatically generated minutes and action item list to participants after the meeting, thereby enabling efficient meetings and effective follow-up.

[1318] The "input meeting schedule and meeting content" is data that the user inputs into the server via the terminal, including detailed information such as the start date and time of the meeting, the location, the topic, and the agenda.

[1319] A "database" is a storage device for efficiently storing and managing detailed information about analyzed meetings, log data, and configuration information for the generation AI.

[1320] "Generative AI" is an artificial intelligence model that asks appropriate questions and provides supplementary explanations based on the content of a meeting, and is a system that operates using specific libraries and algorithms.

[1321] "Live feed" refers to the audio and text data generated in real time during a meeting, which the server provides to the generating AI, allowing the generating AI to participate in the meeting.

[1322] "Log data" refers to a record of all statements made during a meeting and other related data that is later used to create minutes and action items.

[1323] Minutes are documents that summarize the main points of a meeting by organizing what was said, decisions made, and agenda items, and are distributed to participants after the meeting has ended.

[1324] "Action items" are tasks and follow-up matters decided during the meeting that must be completed before the next meeting. By extracting and distributing these to participants, it becomes clear what needs to be completed before the next meeting.

[1325] "Real-time data" refers to the audio and text information generated in real time during a meeting, which is then analyzed by the generating AI.

[1326] The present invention is a system that uses generative AI to improve the efficiency and usefulness of meetings. Specific embodiments for carrying out the present invention will be described in detail below.

[1327] Hardware and Software Use

[1328] This system uses a server, a terminal, and a generative AI model. The terminal is a personal computer or smartphone that communicates with the server. The server has powerful computing power and runs the generative AI model. The following specific software is used:

[1329] Database management systems: MySQL or PostgreSQL

[1330] AI model libraries: TensorFlow, PyTorch

[1331] Communication protocol: HTTPS

[1332] Natural language processing technology: Natural language analysis libraries (e.g., spaCy, NLTK)

[1333] Meeting tool: General online meeting tools (e.g., Zoom or Microsoft Teams)

[1334] System Operation

[1335] The user uses the terminal to input the schedule and content of the meeting. For example, in the case of a meeting about a marketing plan for a new product, the user would input the following:

[1336] Meeting Schedule: October 15, 2023, 14:00

[1337] Meeting topic: New product marketing plan

[1338] The role of generative AI: Supplementary explanation

[1339] When the user clicks the submit button, the input data is sent to the server.

[1340] The server analyzes the received meeting information, converts the meeting details into JSON format, and stores it in a database. Next, it loads the libraries and models required to prepare the generative AI model (e.g., a generative AI model using TensorFlow) and configures it according to the content and roles of the meeting.

[1341] When a meeting starts, the server acquires real-time data from the meeting tool. This data includes audio and text information generated during the meeting. The real-time data is provided to the generation AI, which analyzes what is being said. For example, the generation AI generates appropriate questions and supplementary explanations in real time and outputs them as text through the meeting tool.

[1342] The server also collects all log data during the meeting. After the meeting, minutes are automatically generated based on the log data. The generated minutes include the date and time, participants, agenda, comments, and decisions. The minutes are then distributed to participants via email.

[1343] Furthermore, the server extracts action items from the log data, including next follow-up tasks, and distributes them to participants as a list, allowing for efficient management of tasks decided in the meeting.

[1344] In this way, the invention centralizes meeting efficiency and follow-up management, and by using generative AI, it is possible to maximize the usefulness of meetings.

[1345] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1346] Step 1:

[1347] The user inputs the schedule and details of the meeting from the terminal and sends it.

[1348] A user launches a conference management application using a PC or smartphone.

[1349] In the input form, enter the date, time, location, topic, agenda, and desired role of the generating AI for the meeting.

[1350] Click the submit button to send the input data to the server.

[1351] (Input) Meeting schedule, content, and generation AI role

[1352] (Output) Meeting information sent to the server

[1353] Step 2:

[1354] The server analyzes the received conference information and stores it in a database.

[1355] The server converts the received conference information into JSON format.

[1356] Save the JSON format data to the database.

[1357] (Input) Meeting information received from the user

[1358] (Output) Meeting information stored in the database

[1359] Step 3:

[1360] The server prepares the generative AI model.

[1361] The meeting information is obtained from the database and the generating AI is initialized according to the content and role of the meeting.

[1362] Loading a generative AI model (e.g. using TensorFlow) and required libraries.

[1363] (Input) Meeting information stored in the database

[1364] (Output) Initialized generated AI

[1365] Step 4:

[1366] The server collects real-time data from the meeting and provides it to the generation AI.

[1367] When a meeting starts, the server obtains real-time audio and text data from the meeting tool (e.g., Zoom, Microsoft Teams).

[1368] The acquired real-time data is passed to the generation AI.

[1369] (Input) Real-time data from conferencing tools

[1370] (Output) Real-time data provided to the generative AI

[1371] Step 5:

[1372] The generative AI analyzes the content of statements in real time and generates appropriate questions and supplementary explanations.

[1373] The generative AI analyzes real-time data and generates necessary questions and supplementary explanations based on what is being said and the progress of the meeting.

[1374] The generated questions and supplementary explanations are output to the conference tool and displayed to participants in real time.

[1375] (Input) Real-time data and meeting information

[1376] (Output) Generated questions and supplementary explanations

[1377] Step 6:

[1378] The server collects all log data during the meeting and automatically generates minutes.

[1379] Accumulates log data obtained from conference tools.

[1380] Minutes are automatically generated from the accumulated log data using a minutes generation algorithm.

[1381] The generated minutes include the date, time, participants, agenda, what was said, and what was decided.

[1382] (Input) Log data during the meeting

[1383] (Output) Automatically generated minutes

[1384] Step 7:

[1385] The server distributes the minutes and action item list to participants.

[1386] Automatically generated meeting minutes and a list of action items extracted from the log data.

[1387] The generated minutes and action item list are distributed to participants via email or other means.

[1388] (Input) Automatically generated minutes and action items

[1389] (Output) Meeting minutes and action item list distributed to participants

[1390] The above steps will improve the efficiency and usefulness of meetings.

[1391] (Application example 1)

[1392] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1393] In modern manufacturing, the effectiveness of meetings and their follow-up are important for maximizing factory operational efficiency. However, the traditional manual process of creating meeting minutes and extracting action items requires a great deal of time and effort, resulting in a decline in meeting efficiency. Furthermore, it can be difficult to ask questions or provide supplementary explanations in real time, which can sometimes prevent meetings from proceeding smoothly. To solve this problem, a more efficient meeting support system is needed.

[1394] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1395] In this invention, the server includes means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database, means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the utilization content, and means for a robot located in the factory to monitor the meeting and analyze the progress of the meeting in real time to provide assistance. This increases the effectiveness of meetings and enables the automatic generation of minutes and the extraction and distribution of action items.

[1396] "Meeting schedule" refers to information that includes the date, time, location and associated timeline of a meeting.

[1397] "Meeting content" refers to information including the agenda, topics, and participant roles in a meeting.

[1398] A "database" is an information management system that stores conference details and allows them to be retrieved at a later time.

[1399] "Generative AI" is a system that uses artificial intelligence technology to automatically generate questions and supplementary explanations based on the content of the meeting.

[1400] "Live feed" refers to streaming data that sends real-time information about the progress of a meeting to the generation AI.

[1401] "Questions" are posed by the generative AI in response to comments made or topics discussed during a meeting, and are intended to deepen the discussion.

[1402] "Supplementary explanation" refers to additional information or explanations that make the content of the meeting easier to understand.

[1403] "Log data" refers to digital data that records what was said during a meeting and the progress of the discussion.

[1404] "Minutes" are official documents summarizing the proceedings, statements, and decisions made at a meeting.

[1405] "Action items" are specific tasks or action items that need to be carried out after the meeting ends.

[1406] "Robots" are automated devices that are placed in factories and perform real-time analysis and monitoring of meetings.

[1407] "Monitoring" refers to the act of watching the progress of a meeting and sending that data to a server or generating AI.

[1408] "Real-time analysis" refers to the process of analyzing data collected during a meeting on the spot and having the generating AI immediately take appropriate action.

[1409] This invention is a system that utilizes smart robots to streamline meetings held in factories. This system has the function of inputting the meeting schedule and content and using generative AI to assist the meeting.

[1410] System Configuration

[1411] Hardware:

[1412] Tablet or smartphone: A device that allows users to enter meeting schedules and details.

[1413] Server: Handles database and generation AI processing.

[1414] Smart robots: Deployed in factories, they monitor meetings and provide real-time analysis.

[1415] software:

[1416] Django or Flask: for server-side data processing and secure communication.

[1417] TensorFlow and PyTorch: For running and training generative AI models.

[1418] OpenCV: For robotic video and audio capture and analysis.

[1419] What the program does

[1420] 1. User Input:

[1421] Users input the meeting schedule (date, time, location) and content (agenda, participants) via a tablet or smartphone, as well as requests to the generation AI (questioners, supplementary explanations, etc.). This information is sent to the server via a dedicated web form.

[1422] 2. Prepare the generative AI:

[1423] The server analyzes the received meeting details and stores them in a database. Based on this information, it initializes a generative AI model and inputs the meeting content and roles. Here, TensorFlow or PyTorch are used to train and run the generative AI.

[1424] 3. Real-time meeting participation:

[1425] When a meeting begins, a smart robot located within the factory monitors the meeting. The robot uses libraries such as OpenCV to capture video and audio data and send it to a server. The server's generative AI analyzes this data in real time and asks appropriate questions or provides additional explanations.

[1426] 4. Automatic generation of meeting minutes:

[1427] After the meeting ends, the server automatically generates minutes based on the log data collected during the meeting. These minutes include the progress of the meeting, what was said, and what decisions were made. The generated minutes are automatically distributed to all participants.

[1428] 5. Extract action items:

[1429] The server extracts next action items from the generated minutes and organizes them into a list, which is also automatically distributed to all participants.

[1430] Specific examples

[1431] Below is an example of a meeting, along with the minutes and action items generated from it:

[1432] Prompt Sentence Examples

[1433] Meeting on new product marketing plan

[1434] Date and time: October 15, 2023 14:00

[1435] Location: Factory Conference Room B

[1436] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[1437] Agenda: New product launch plans, advertising strategies, sales forecasts

[1438] Generative AI role: Supplementary explanation

[1439] Example of generated meeting minutes and action items

[1440] minutes

[1441] Meeting Date and Time: October 15, 2023, 14:00

[1442] Location: Factory Conference Room B

[1443] Participants: Factory Manager A, Factory Manager B, Factory Manager C

[1444] Agenda: New product launch plans, advertising strategies, sales forecasts

[1445] Statement:

[1446] Administrator A: The new product is scheduled to go on sale in November.

[1447] Administrator B: The advertising campaign is scheduled to begin in late October.

[1448] Manager C: Our sales forecast is for 10,000 units per year.

[1449] Action Items

[1450] Administrator A: Preparation for sales launch in November

[1451] Administrator B: Preparing for advertising campaign in late October

[1452] Manager C: Sales forecast specifics and schedule adjustments

[1453] As in this example, the generated minutes and action items allow for efficient follow-up of the meeting.

[1454] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1455] Step 1:

[1456] Users use tablets or smartphones to enter meeting schedules and details. The entered data, including details such as date, time, location, agenda, and participants, is sent to the server via a web form. This information is then stored in a database on the server side.

[1457] Step 2:

[1458] The server analyzes the saved meeting details and prepares the generative AI. Specifically, it initializes the generative AI model (such as TensorFlow or PyTorch) based on the meeting content and roles, and loads the necessary libraries and models. The analysis results in meeting summary data.

[1459] Step 3:

[1460] When a meeting begins, smart robots deployed within the factory capture video and audio and send it to a server. This captured data is processed using libraries such as OpenCV and sent to the server as a live feed of the meeting in real time. The server then provides this input data to the generative AI.

[1461] Step 4:

[1462] The generation AI analyzes the live feed data in real time. On the server side, the generation AI generates questions and supplementary explanations appropriate for the progress of the meeting based on the input live feed data. This creates output messages according to the timing of speech.

[1463] Step 5:

[1464] During the meeting, questions and supplementary explanations posed by the generation AI are provided to meeting participants through the robot. The robot outputs audio and presents the generated messages during the meeting. This process helps the meeting discussion proceed smoothly.

[1465] Step 6:

[1466] Once the meeting is over, the server automatically generates minutes based on the log data collected during the meeting. These minutes are generated using Natural Language Processing (NLP) technology and output as a document containing the meeting history, statements made, decisions made, and more.

[1467] Step 7:

[1468] The server extracts next action items from the generated minutes and organizes them into a list. This list is also analyzed using NLP technology to generate an action item list that clearly indicates specific tasks and responsible persons.

[1469] Step 8:

[1470] Finally, the generated minutes and action item list are automatically distributed to all participants, primarily via email or an internal company sharing system, allowing participants to refer to the list and efficiently plan the next meeting or follow-up work.

[1471] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1472] The present invention is a system that combines generative AI and an emotion engine to improve the effectiveness of meetings, and detailed embodiments thereof will be described below.

[1473] Input of meeting schedule and content

[1474] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[1475] Preparing for generative AI and emotion engines

[1476] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[1477] Generative AI meeting participation

[1478] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[1479] Taking meeting minutes

[1480] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[1481] Extracting action items

[1482] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[1483] Specific examples

[1484] A concrete example is a meeting about a marketing plan for a new product.

[1485] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[1486] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion engine and initializes the user emotion recognition function.

[1487] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[1488] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[1489] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[1490] By combining generative AI with an emotion engine, the system of the present invention further improves the effectiveness of meetings, automates the creation of meeting minutes and the organization of action items, and simplifies post-meeting follow-up.

[1491] The processing flow will be explained below.

[1492] Step 1:

[1493] The user enters the meeting schedule and details from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[1494] Step 2:

[1495] The server receives and analyzes the data sent by the user, specifically checking information such as the meeting date, time, location, topic, agenda, and AI role, and converts it into a format that is easy to understand.

[1496] Step 3:

[1497] The server stores the formatted meeting details in a database, which manages details such as the meeting schedule, content, participant list, and the role of the generating AI.

[1498] Step 4:

[1499] The server initializes the generated AI, loading the necessary libraries and AI models, and allocating memory and CPU / GPU resources to prepare an instance of the generated AI.

[1500] Step 5:

[1501] The server inputs meeting details into the generation AI, specifically providing it with the meeting topic, set roles, and pre-set content.

[1502] Step 6:

[1503] The server initializes the emotion engine and prepares the emotion recognition module, which includes functions for recognizing the user's face, voice analysis, and text analysis.

[1504] Step 7:

[1505] The server streams the live feed (real-time data of the meeting) to the generation AI and emotion engine. When the meeting starts, real-time data such as audio, text, and video is provided to the generation AI and emotion engine.

[1506] Step 8:

[1507] The generative AI analyzes the live feed and asks appropriate questions as the meeting progresses, providing additional explanations based on emotions, etc. The emotion engine monitors changes in the user's emotions in real time and feeds that information back to the generative AI.

[1508] Step 9:

[1509] The server collects log data and emotion data during the meeting. It records and stores data on all statements, discussions, decisions, and important emotion changes during the meeting.

[1510] Step 10:

[1511] The server automatically generates minutes based on the collected log data and emotion data. The minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are then reviewed to check for errors or omissions.

[1512] Step 11:

[1513] The server distributes the generated minutes to participants after the meeting ends, via email or a dedicated system.

[1514] Step 12:

[1515] The server extracts next action items based on the meeting log data and emotion data. These action items list follow-up tasks and activities to be carried out before the next meeting.

[1516] Step 13:

[1517] The server distributes the generated action item list to participants after the meeting, improving follow-up efficiency and simplifying task management.

[1518] Example 2

[1519] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1520] In conventional conferencing systems, meeting content and progress management is often done manually, which tends to reduce meeting efficiency. It is also difficult to consider changes in participants' emotions during the meeting, which can lead to inappropriate questions or supplementary explanations not being given. Furthermore, creating meeting minutes and extracting action items takes time, which makes follow-up ineffective.

[1521] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1522] In this invention, the server includes a means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a storage device, a means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content, and a means for preparing an emotion recognition engine and collecting user emotion data. This improves meeting efficiency and enables appropriate questions and supplementary explanations based on participants' emotions. Furthermore, the server can automatically generate minutes and extract and distribute next action items, enabling effective follow-up.

[1523] "Meeting schedule" refers to detailed information about a meeting, such as the date, time, location, participants, and agenda.

[1524] "Meeting content" refers to information including the topics or agenda to be discussed at the meeting, the procedure, and any necessary materials or information.

[1525] "Storage device" refers to a device or system for storing digital information, such as a database.

[1526] "Generative AI" refers to an artificial intelligence model that automatically generates questions and follow-up explanations as a meeting progresses.

[1527] "Emotion recognition engine" refers to a module that analyzes a user's voice, video, and text data to recognize their emotional state.

[1528] "Live Feed" means the stream of audio, video, and text data provided in real time during a conference.

[1529] "Log data" refers to data such as participants' comments, actions, and timestamps that are recorded during the meeting.

[1530] "Minutes" refers to an official record document that summarizes the progress of a meeting, what was said, and what decisions were made.

[1531] "Action items" refer to specific follow-up tasks or activities decided upon at the meeting.

[1532] This invention is a meeting support system that uses generative AI and an emotion recognition engine to improve the efficiency of meetings and simplify the automatic creation of minutes and management of follow-up tasks.

[1533] Input of meeting schedule and content

[1534] The user uses a device (PC or smartphone) to enter the meeting schedule and details into a dedicated input form. Specifically, the user fills in the form on the PC or smartphone with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, "questioner" or "supplementary explanation"), and clicks the send button.

[1535] Preparing for generative AI and emotion recognition engines

[1536] The server analyzes the conference information received from the user and stores it in a database. Specifically, it analyzes the received data in JSON format, extracts the necessary information, and records it in the database.

[1537] Next, the server prepares the generation AI. For example, the AI ​​model used here is OpenAI's GPT-4. The server provides the generation AI with a prompt based on the meeting content and initializes the generation AI.

[1538] The server also configures an emotion recognition engine. It installs facial recognition, voice analysis, and text analysis modules and configures them to recognize user emotions in real time. For example, we use the Microsoft Azure Emotion Analysis API.

[1539] Generative AI meeting participation

[1540] When the meeting begins, the server provides a live feed to the generation AI, including audio, video, and text data from the meeting. The generation AI analyzes the provided live feed and asks appropriate questions or provides supplementary explanations based on the user's emotional data. Changes in the user's emotions are monitored by an emotion recognition engine, and the data is also fed back to the generation AI in real time.

[1541] Taking meeting minutes

[1542] During the meeting, the server collects all log data, including audio, video, text, and emotion data. After the meeting ends, the server uses generative AI to create minutes based on the collected log data. The generated minutes include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The minutes are distributed to all participants via email or chat tools.

[1543] Extracting action items

[1544] The server analyzes the meeting log data and emotion data to extract action items to be discussed at the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting. This streamlines the management of follow-up tasks decided in the meeting.

[1545] Specific examples

[1546] A concrete example is a meeting about a marketing plan for a new product.

[1547] 1. The user inputs the meeting schedule (October 15, 2023, 2:00 PM), the meeting content (marketing plan for a new product), and the desired role of the generative AI (supplementary explanation) on the device.

[1548] 2. The server receives the information, initializes the generation AI, and inputs the meeting content into the AI. At the same time, it prepares the emotion recognition engine and initializes the user emotion recognition function.

[1549] 3. Once the meeting begins, the generative AI analyzes the live feed of the meeting and provides appropriate follow-up explanations and questions based on the user's emotions.

[1550] 4. The server collects log data and emotion data and automatically generates minutes, which are distributed to all participants after the meeting.

[1551] 5. The server creates a list of next action items based on the meeting log data and emotion data and distributes them to participants.

[1552] Example prompts for generative AI models

[1553] "A user has scheduled a meeting to discuss a marketing plan for a new product. The meeting will start at 14:00 on October 15, 2023, and the generative AI will act as a supplementary explanation. Analyze the user's emotions in real time and ask appropriate questions or provide supplementary explanations based on the data from the emotion engine."

[1554] As described above, the system of the present invention can improve the efficiency of meetings and automate the creation of minutes and the management of action items.

[1555] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1556] Step 1:

[1557] The user inputs the conference information from the terminal.

[1558] The input details include the date and time (e.g., 15 October 2023, 2:00 PM), location (e.g., Conference Room A), topic (e.g., marketing plan for a new product), agenda (e.g., sharing market analysis results), and desired role of the generative AI (e.g., supplementary explanation). Specifically, users enter this information into the input form on their PC or smartphone and click the send button.

[1559] Input: Meeting date, time, location, topic, agenda, desired role of the AI

[1560] Output: Data sent from the device to the server

[1561] Step 2:

[1562] The device sends the input information to the server. The device uses an HTTP POST request to send the input data to the server's API endpoint.

[1563] Input: User-entered meeting information

[1564] Output: Meeting information data sent to the server

[1565] Step 3:

[1566] The server parses the received data. The server parses the meeting information in JSON format, extracts the necessary fields (date, time, topic, role, etc.), and stores them in the database. Specifically, it parses the data and stores the extracted information in the appropriate table.

[1567] Input: Meeting information data sent from the terminal

[1568] Output: Parsed meeting information, data stored in a database

[1569] Step 4:

[1570] The server initializes the generation AI, loads OpenAI's GPT-4 model, and provides the AI ​​with prompts based on the meeting content.

[1571] Input: Parsed meeting information (e.g., meeting date and time, new product marketing plan, additional explainers)

[1572] Output: Initialized generated AI

[1573] Step 5:

[1574] The server configures the emotion recognition engine. The emotion recognition engine installs facial recognition, voice analysis, and text analysis modules and is configured to recognize user emotions in real time. For example, the Microsoft Azure emotion analysis API is used.

[1575] Input: Server configuration information, emotion recognition engine

[1576] Output: Initialized emotion recognition engine

[1577] Step 6:

[1578] When a meeting starts, the server provides a live feed to the generation AI, capturing audio, video, and text data from the meeting in real time and sending it to the generation AI.

[1579] Input: Live meeting feed (audio, video, text)

[1580] Output: Live feed data provided to the generative AI

[1581] Step 7:

[1582] The generative AI analyzes the live feed and asks appropriate questions or provides additional explanations based on the user's emotional data. Emotional data from the emotion recognition engine is also collected and fed back to the generative AI.

[1583] Input: Live feed data, emotion data

[1584] Output: appropriate questions and further explanations

[1585] Step 8:

[1586] The server collects log data during the meeting, recording the meeting log in real time, including audio, video, text, and emotional data, and storing it in a database.

[1587] Input: Contents of remarks made during meetings, emotional data

[1588] Output: Collected and stored log data

[1589] Step 9:

[1590] After the meeting ends, minutes are created using a generation AI based on the log data collected by the server.

[1591] Input: Collected log data

[1592] Output: Generated meeting minutes

[1593] Step 10:

[1594] The server distributes the generated minutes to all participants via email or chat tools. Specifically, the minutes file is attached to an email and sent to the participants.

[1595] Input: Generated minutes

[1596] Output: Minutes distributed to participants

[1597] Step 11:

[1598] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[1599] Input: Meeting log data, emotion data

[1600] Output: Extracted action items list

[1601] Step 12:

[1602] The server distributes the list of action items to participants by attaching the list to an email with a message saying, "Please complete the following action items before the next meeting."

[1603] Input: Extracted action item list

[1604] Output: Action item list distributed to participants

[1605] (Application example 2)

[1606] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1607] Conventional conferencing systems provide insufficient support for improving meeting efficiency, and it is difficult to conduct meetings while taking into account the emotions of employees, especially in environments such as smart factories. Furthermore, creating meeting minutes and extracting action items requires a lot of time and effort.

[1608] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for analyzing the input meeting schedule and meeting content and saving detailed information about the meeting in a database; means for preparing a generation AI based on the saved detailed information about the meeting, setting roles, and inputting the meeting content; means for providing a live feed of the meeting to the generation AI and having the generation AI ask appropriate questions and provide supplementary explanations based on the usage content; means for collecting log data during the meeting, automatically generating minutes, and distributing them to participants; means for extracting next action items from the meeting log data and distributing them to participants; and means for analyzing employee emotions during the meeting in real time and feeding the emotion data back to the generation AI. This enables optimization of meeting progress through real-time emotion analysis and automatic generation of minutes and action items after the meeting ends.

[1609] A "meeting schedule" is planning information about the dates, times, locations, and topics of meetings.

[1610] "Meeting content" refers to information such as the agenda and topics to be discussed during the meeting, as well as planned remarks and materials.

[1611] A "database" is a system for storing and managing data in a structured way.

[1612] "Generative AI" is artificial intelligence that automatically generates text, questions, answers, etc. based on input information.

[1613] A "live feed" is a continuous stream of data, such as audio, video, or text, provided in real time.

[1614] "Emotional data" refers to information about an employee's emotions obtained through facial expressions and voice analysis.

[1615] Minutes are a document that records the statements and decisions made during a meeting.

[1616] "Action items" are follow-up tasks or next agenda items decided at the meeting.

[1617] This invention is a system that improves the effectiveness of meetings by combining a generative AI and an emotion engine. Specific embodiments for carrying out the invention will be described below.

[1618] 1. Input of meeting schedule and content

[1619] The user inputs the meeting schedule and details into the server from a device (PC or smartphone). The user fills in the input form with the meeting date, time, location, topic, agenda, and the role desired for the generation AI (for example, questioner or supplementary explanation), and clicks the send button.

[1620] 2. Preparing the generative AI and emotion engine

[1621] The server analyzes the received data and stores it in a database. It then initializes the generative AI based on the stored information, loads the necessary libraries and models, and prepares the emotion engine, installing and configuring the facial recognition, voice analysis, and text analysis modules to recognize the user's emotions.

[1622] In particular, the emotion engine uses OpenCV to detect faces from video feeds and analyze facial expressions, natural language processing libraries such as Spacy for speech analysis, and the generative AI uses the Transformers library from Hugging Face.

[1623] 3. Generative AI participation in meetings

[1624] The server provides the user's real-time emotional data to the generation AI through the emotion engine. When the meeting begins, the generation AI analyzes the live feed of the meeting (audio, video, text) and asks appropriate questions or provides supplementary explanations based on the user's emotions. The emotion engine monitors the user's emotional changes in real time and feeds that information back to the generation AI.

[1625] 4. Taking meeting minutes

[1626] The server collects log data during the meeting and automatically generates minutes, which include the meeting date and time, participants, agenda, comments, decisions, and notes on important emotional changes. The generated minutes are distributed to participants after the meeting ends.

[1627] 5. Extracting action items

[1628] The server analyzes the meeting log data and emotion data to extract next action items. The extracted action items are organized into a list and distributed to participants after the meeting ends. This allows for efficient management of follow-up tasks decided in the meeting.

[1629] Prompt Sentence Examples

[1630] "In today's meeting, we discussed the maintenance schedule and operational issues for a new product line. Please analyze what was said and extract important action items based on the following information."

[1631] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1632] Step 1:

[1633] Users input meeting schedules and content

[1634] Using a device (PC or smartphone), the user fills in the input form with the date, time, location, agenda, topic, and desired role of the generating AI (e.g., questioner or supplementary explanation), and clicks the send button. The input data is sent to the server.

[1635] Input: Meeting schedule information (date, time, location), meeting content (agenda, topic, role of the generation AI)

[1636] Output: Meeting schedule and contents saved on the server

[1637] Step 2:

[1638] The server analyzes the meeting information and stores it in a database.

[1639] The server parses the received meeting schedule and content data, extracts the necessary information, and stores it in a database, preparing the data for future reference and for use in initializing the generative AI.

[1640] Input: Meeting schedule and content data submitted by users

[1641] Output: Parsed meeting information stored in a database

[1642] Step 3:

[1643] The server prepares the generative AI and emotion engine

[1644] Based on the saved meeting information, the server initializes the generative AI, loads the necessary libraries and models (e.g., Hugging Face Transformers), prepares the emotion engine (e.g., OpenCV or Spacy), and installs and configures the facial recognition, speech analysis, and text analysis modules.

[1645] Input: Meeting information stored in the database

[1646] Output: Initialized generative AI, configured emotion engine

[1647] Step 4:

[1648] The server provides a live feed to the generation AI, feeding back emotional data.

[1649] When a meeting begins, the server provides a live feed of the meeting (audio, video, and text data) to the Generator AI, which analyzes the data and asks appropriate questions or provides additional explanations. The Emotion Engine analyzes employees' emotions in real time and provides feedback on the emotion data to the Generator AI.

[1650] Input: Live feed (audio, video, text data), emotion data

[1651] Output: Appropriate questions and supplementary explanations from generative AI

[1652] Step 5:

[1653] The server collects meeting log data and automatically generates minutes.

[1654] The server collects audio, video, and text data from the meeting as logs, and then, using pre-set rules and the power of generative AI, automatically creates minutes of the meeting and distributes them to all participants after the meeting has ended.

[1655] Input: Audio, video, and text data during the meeting

[1656] Output: Generated meeting minutes

[1657] Step 6:

[1658] The server extracts the action items and distributes them to the participants.

[1659] The server analyzes the meeting log data and emotion data to extract action items for the next meeting. The extracted action items are organized into a list and distributed to participants after the meeting ends.

[1660] Input: Meeting log data, emotion data

[1661] Output: Action item list

[1662] By dividing the specific processing flow into steps in this way, we aim to make the "Form for carrying out the invention" as a patent specification more clearly understandable.

[1663] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1664] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1665] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1666] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1667] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1668] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1669] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1670] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1671] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1672] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1673] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1674] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1675] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1676] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1677] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1678] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1679] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1680] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1681] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1682] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1683] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1684] The following is further disclosed regarding the above embodiment.

[1685] (Claim 1)

[1686] A means for analyzing the input meeting schedule and meeting content and storing detailed information of the meeting in a database;

[1687] A means to prepare the generative AI based on the saved meeting details, set roles, and input the meeting content;

[1688] A means to provide a live feed of the meeting to the Generative AI, which can then ask appropriate questions or provide additional explanations based on the content it uses.

[1689] A means for collecting log data during a meeting, automatically generating minutes, and distributing them to participants;

[1690] A method to extract next action items from the meeting log data and distribute them to participants.

[1691] A system including:

[1692] (Claim 2)

[1693] The system of claim 1 further comprises a means for the generating AI to analyze the content of statements in real time during the meeting and ask questions or provide additional explanations as necessary.

[1694] (Claim 3)

[1695] 10. The system of claim 1, further comprising means for automatically distributing the generated minutes and action item list to participants after the meeting is over.

[1696] "Example 1"

[1697] (Claim 1)

[1698] A means for analyzing the input meeting schedule and meeting content and storing detailed information of the meeting in a database;

[1699] A means to prepare the generative AI based on the saved meeting details, set roles, and input the meeting content;

[1700] A means to provide a live feed of the meeting to the Generative AI, which can then ask appropriate questions or provide additional explanations based on the content it uses.

[1701] A means for collecting log data during a meeting, automatically generating minutes, and distributing them to participants;

[1702] A method to extract next action items from the meeting log data and distribute them to participants.

[1703] A means to obtain real-time data from the conference tool and provide it to the generation AI,

[1704] A means for the generation AI to analyze the content of statements in real time and generate questions and supplementary explanations,

[1705] A means to distribute automatically generated minutes and action item lists to participants after the meeting;

[1706] A system including:

[1707] (Claim 2)

[1708] The system of claim 1 further comprises a means for the generating AI to analyze the content of statements in real time during the meeting and ask questions or provide additional explanations as necessary.

[1709] (Claim 3)

[1710] 10. The system of claim 1, further comprising means for automatically distributing the generated minutes and action item list to participants after the meeting is over.

[1711] "Application Example 1"

[1712] (Claim 1)

[1713] A means for analyzing the input meeting schedule and meeting content and storing detailed information of the meeting in a database;

[1714] A means to prepare the generative AI based on the saved meeting details, set roles, and input the meeting content;

[1715] A means to provide a live feed of the meeting to the Generative AI, which can then ask appropriate questions or provide additional explanations based on the content it uses.

[1716] A means for collecting log data during a meeting, automatically generating minutes, and distributing them to participants;

[1717] A method to extract next action items from the meeting log data and distribute them to participants.

[1718] Robots placed in factories monitor meetings and provide assistance by analyzing the progress of meetings in real time.

[1719] A system including:

[1720] (Claim 2)

[1721] The system of claim 1 further comprises a means for the generating AI to analyze the content of statements in real time during the meeting and ask questions or provide additional explanations as necessary.

[1722] (Claim 3)

[1723] 10. The system of claim 1, further comprising means for automatically distributing the generated minutes and action item list to participants after the meeting is over.

[1724] "Example 2: Combining Emotion Engines"

[1725] (Claim 1)

[1726] means for analyzing the input conference schedule and conference content and storing detailed information of the conference in a storage device;

[1727] A means for preparing a generative AI based on saved meeting details, setting roles, and inputting meeting content;

[1728] providing an emotion recognition engine and means for collecting user emotion data;

[1729] A means to provide a live feed of the meeting to the Generative AI, which can then ask appropriate questions or provide additional explanations based on the content it uses.

[1730] A means to collect log data and emotion data during meetings, automatically generate minutes, and distribute them to participants;

[1731] A method to extract next action items from meeting log data and emotion data and distribute them to participants.

[1732] A system including:

[1733] (Claim 2)

[1734] The system of claim 1 further comprises a means for the generating AI to analyze the content of statements and the user's emotions in real time during the meeting and to ask questions or provide additional explanations as necessary.

[1735] (Claim 3)

[1736] 10. The system of claim 1, further comprising means for automatically distributing the generated minutes and action item list to participants after the meeting is over.

[1737] "Application example 2 when combining emotion engines"

[1738] (Claim 1)

[1739] A means for analyzing the input meeting schedule and meeting content and storing detailed information of the meeting in a database;

[1740] A means to prepare the generative AI based on the saved meeting details, set roles, and input the meeting content;

[1741] A means to provide a live feed of the meeting to the Generative AI, which can then ask appropriate questions or provide additional explanations based on the content it uses.

[1742] A means for collecting log data during a meeting, automatically generating minutes, and distributing them to participants;

[1743] A method to extract next action items from the meeting log data and distribute them to participants.

[1744] A method to analyze the emotions of employees during meetings in real time and feed that emotional data back to the generating AI,

[1745] A system including:

[1746] (Claim 2)

[1747] The system of claim 1 further comprises a means for the generating AI to analyze the content of statements in real time during a meeting and ask appropriate questions or provide supplementary explanations based on the emotions of employees.

[1748] (Claim 3)

[1749] 10. The system of claim 1, further comprising means for automatically distributing the generated minutes and action item list to participants after the meeting is over.

[1750] (Claim 4)

[1751] The system of claim 1 further comprises a means for the generation AI to ask questions and provide supplementary explanations as the meeting progresses, and to utilize the results of real-time emotion analysis when necessary. [Explanation of symbols]

[1752] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing the input meeting schedule and meeting content and storing detailed information of the meeting in a database; A means to prepare the generative AI based on the saved meeting details, set roles, and input the meeting content; A means to provide a live feed of the meeting to the Generative AI, which can then ask appropriate questions or provide additional explanations based on the content it uses. A means for collecting log data during a meeting, automatically generating minutes, and distributing them to participants; A method to extract next action items from the meeting log data and distribute them to participants. A system including:

2. The system according to claim 1, further comprising a means for the generating AI to analyze the content of statements in real time during a meeting and to ask questions or provide supplementary explanations as necessary.

3. The system of claim 1 , further comprising means for automatically distributing the meeting minutes and action item list generated after the meeting is over to the participants.

Images