System

Abstract

To provide a system for improving the quality of a conference, preventing a problem from being overlooked, promoting active discussion, and improving business efficiency.SOLUTION: Means for inputting a meeting schedule and an agenda, means for receiving minutes of a previous meeting and relevant materials, means for storing the received information, means for inputting the minutes of the previous meeting and the agenda to a generation AI and generating questions and supplementary information, means for notifying the start of the meeting, means for converting meeting content into text in real time using speech recognition technology, means for monitoring the progress of the meeting in real time by the generation AI and providing appropriate questions and supplements, means for providing real-time translations of the meeting content by the generation AI, means for creating minutes and summarizing next action items by the generation AI after the end of the meeting, and means for notifying the created minutes and the summary of the next action items.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Meetings often involve situations where no one asks questions or problems are overlooked. In such cases, the meeting simply ends smoothly, but no fruitful discussions or decisions are made. Furthermore, creating meeting minutes and summarizing action items for the next meeting is time-consuming and reduces productivity. Furthermore, in meetings where language barriers exist, communication is often difficult. This invention aims to solve these problems, making meetings more fulfilling and improving productivity. [Means for solving the problem]

[0005] This invention improves the quality of meetings, prevents issues from being overlooked, promotes lively discussions, and achieves work efficiency through a system that includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for storing the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; and means for using the saved data to prepare for the next meeting. Furthermore, by including means for the generation AI to pre-set multiple roles to participate in the meeting and means for the generation AI to automatically format the minutes and summaries of next action items, even more efficient meeting management is possible.

[0006] "Meeting schedule" is information that indicates the date, time, location, and purpose of a meeting scheduled by participants.

[0007] An "agenda" is a list of items or topics to be discussed at a meeting.

[0008] "Generative AI" refers to artificial intelligence that uses natural language processing and machine learning technologies to provide supplementary explanations for meeting content, generate questions, and create minutes.

[0009] "Last Meeting Minutes" are documents that describe what was discussed, decisions made, and action items taken at the last meeting.

[0010] "Related materials" refers to reference documents and data related to the meeting, and is information to support the discussion.

[0011] "Voice recognition technology" is a technology that converts voice data into text.

[0012] "Real-time text conversion" refers to the use of speech recognition technology to instantly convert speech into text format in real time.

[0013] "Questions and supplementary information" is information created by generative AI during or before a meeting to help deepen discussions and understanding.

[0014] "Real-time translation of meeting content" is the process of translating what is said during a meeting into different languages ​​in real time.

[0015] Minutes are a document that records the progress of a meeting, decisions made, and next action items.

[0016] "Next action items" are specific tasks or activities that should be carried out after the meeting ends.

[0017] "Notification" refers to the system sending messages to users informing them of important information or updates.

[0018] "Stored data" refers to information such as meeting contents, materials, and minutes that are stored in a database and made available for later use.

[0019] "Roles" refers to the function of pre-setting what the generative AI will do in a meeting, and specifically includes roles such as questioner, supplementary information provider, and minutes writer.

[0020] "Format" refers to the format in which the generating AI automatically organizes minutes and next action items and outputs them in an easy-to-read format. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The system of this invention provides a series of functions for enhancing conferences through collaboration between the server, terminals, and users. Below, we will explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[0043] overview

[0044] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for questions and supplementary information, real-time translation, meeting minutes, and a summary of upcoming action items.

[0045] System execution example

[0046] 1. Preparation Phase

[0047] The user inputs the meeting schedule and specific agenda into the system. For example, the user inputs "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0048] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[0049] This information is stored in a database by the server.

[0050] 2. Before the meeting begins

[0051] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into the generative AI model.

[0052] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[0053] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[0054] 3. During a meeting

[0055] When a meeting begins, the user tells the system to "start the meeting."

[0056] The device recognizes the audio of the meeting in real time and converts it into text.

[0057] The server monitors the progress of the meeting in real time with a generative AI. For example, if a new product software is being discussed, the generative AI will ask appropriate questions, such as, "Could you please tell me more about the status of testing of the new feature?"

[0058] The user's statements are recorded, and the generating AI provides supplementary information as needed.

[0059] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[0060] 4. After the meeting

[0061] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0062] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[0063] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[0064] Specific examples

[0065] For example, if we take a new product release planning meeting, the flow would be as follows:

[0066] 1. A user schedules a new product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00.

[0067] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "marketing plan" to the system.

[0068] 3. The server obtains this information and inputs it into the generative AI model, which generates questions and supplementary information in advance.

[0069] 4. The conference begins and the user notifies the system to "start the conference."

[0070] 5. The device converts the audio of the meeting into text, and the server uses generated AI to provide appropriate questions and additional information.

[0071] 6. The device performs real-time translation, enabling multilingual support.

[0072] 7. After the meeting, the server uses AI to create minutes and summarize next action items.

[0073] 8. The server notifies the user of the minutes and next action items and stores them in the database.

[0074] In this way, the AI-Meeting Assistant system provides comprehensive support from meeting preparation to progress and follow-up after the meeting has ended, helping to ensure fruitful meetings.

[0075] The processing flow will be explained below.

[0076] Step 1:

[0077] A user logs in to the system and enters the schedule and agenda for a meeting. For example, the user enters, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0078] Step 2:

[0079] A user uploads the minutes of a previous meeting and related documents to the system, such as a specification document for a new product or a marketing plan.

[0080] Step 3:

[0081] The server receives the meeting schedule, agenda, and related materials sent by the user, and then categorizes and stores the information in a database.

[0082] Step 4:

[0083] The server retrieves the saved meeting agenda, previous meeting minutes, and related materials from the database and inputs them into the generative AI model.

[0084] Step 5:

[0085] The generative AI analyzes the meeting content and generates important questions and supplementary information in advance, which is then stored on the server.

[0086] Step 6:

[0087] The server assigns roles to the generation AI, such as "questioner," "supplementary information provider," and "minutes creator."

[0088] Step 7:

[0089] When it is time for the meeting to start, the user notifies the system that the meeting should begin, for example by clicking a button.

[0090] Step 8:

[0091] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[0092] Step 9:

[0093] The server monitors the progress of the meeting in real time through a generative AI, which then asks appropriate questions to participants, such as, "Please tell me about the status of testing the software for our new product."

[0094] Step 10:

[0095] Users can answer questions posed by the AI, further deepening the discussion, and the AI ​​can provide additional information as needed.

[0096] Step 11:

[0097] The device translates the meeting content in real time, allowing participants who speak different languages ​​to understand the content.

[0098] Step 12:

[0099] After the meeting, the server uses the AI ​​to create minutes based on the audio and text data of the meeting. For example, the minutes could read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0100] Step 13:

[0101] The generative AI summarizes the next action items discussed during the meeting, listing specific tasks such as "finish marketing materials by the next meeting."

[0102] Step 14:

[0103] The server notifies the user of the created minutes and a summary of the next action items, for example, via email or in-system notifications.

[0104] Step 15:

[0105] Use the meeting information, minutes, and summary of next action items stored in the database to prepare for the next meeting, making it more effective.

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] In conventional conferencing systems, meeting preparation, proceeding, and minutes creation are done manually, which requires time and effort. Furthermore, difficulties with real-time speech recognition and translation, and the provision of appropriate questions and supplementary information, result in a decline in meeting efficiency and quality. This invention aims to solve these problems by automating meeting preparation, proceeding, minutes creation, and summarizing next action items.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes means for inputting meeting schedules and agendas, means for receiving minutes and related materials from the previous meeting, means for storing the received information, means for inputting the minutes and agenda from the previous meeting into a generative AI model and generating questions and supplementary information, means for notifying the start of the meeting, means for converting the meeting content into text in real time using speech recognition technology, means for the generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information, means for the generative AI model to provide real-time translation of the meeting content, means for the generative AI model to create minutes and summarize next action items after the meeting ends, means for notifying the created minutes and summaries of next action items, and means for using the stored data to prepare for the next meeting. This makes it possible to streamline a series of meeting processes, such as preparing for the meeting, conducting the meeting, creating minutes, and summarizing next action items, thereby improving the quality and productivity of meetings.

[0111] A "meeting schedule and agenda entry means" is an interface through which a user enters meeting dates, times, locations, and agendas into the system.

[0112] The "means for receiving minutes of previous meetings and related materials" is an interface that allows users to upload records of previous meetings and documents related to meetings to the system.

[0113] The "means for storing received information" is a function for storing the meeting schedule and agenda entered by the user and the uploaded materials in a database.

[0114] "Means of inputting previous meeting minutes and agendas into a generative AI model to generate questions and follow-up information" refers to the process and techniques for inputting stored meeting data into a generative AI model to generate important questions and follow-up information.

[0115] The "means for notifying the start of a conference" is an operation interface that allows a user to notify the system of the start of a conference.

[0116] "Means for converting meeting content into text in real time using speech recognition technology" refers to technology and devices for converting speech made during a meeting from voice to text in real time.

[0117] "Means for a generative AI model to monitor the progress of a meeting in real time and provide appropriate questions and supplementary information" refers to a technology that enables a generative AI model to analyze the content of a meeting, ask appropriate questions on the spot, and provide supplementary information.

[0118] "Means for providing real-time translation of meeting content using generative AI models" refers to technologies and devices that translate speech in real time during a meeting so that all participants who speak different languages ​​can understand the content.

[0119] "Means for a generative AI model to create minutes after a meeting and summarize action items for the next meeting" refers to a technology that analyzes the contents of a meeting from recorded audio and text data, and generates minutes and action items to be addressed at the next meeting.

[0120] The "means for notifying the user of the created minutes and a summary of the next action item" is a function for notifying the user of the created minutes and action items.

[0121] The "means for utilizing stored data in preparation for the next meeting" refers to the process and technology for utilizing stored meeting data in preparation for the next meeting.

[0122] The system of the present invention provides a series of functions for efficiently proceeding and managing a conference through collaboration between a server, terminals, and users. An embodiment of the program of this system will be described in detail below.

[0123] System Configuration

[0124] server

[0125] The server plays a central role in the system and provides the following functions:

[0126] 1. Database management: Store, retrieve, and manage meeting schedules, agendas, minutes, related materials, etc. MySQL or PostgreSQL is used as the RDBMS.

[0127] 2. Generative AI model interface: Input data into a generative AI model (e.g., OpenAI GPT-4) and have it generate questions and supplemental information. The generative AI model operates by receiving the following prompts:

[0128] "Generate important questions and supporting information to discuss at the next meeting."

[0129] 3. Real-time data processing: Audio and text data from meetings are processed in real time and passed to the generative AI model.

[0130] Terminal

[0131] The device is used at the meeting site and provides the following features:

[0132] 1. Speech Recognition: Converts meeting audio into text in real time using Google Cloud Speech-to-Text and Azure Speech Services.

[0133] 2. Real-time translation: Translates meeting content in real time, enabling multilingual support. Translation is performed using the Google Translate API and DeepL API.

[0134] 3. User operation interface: Provides a UI for users to start meetings and upload materials, using web forms or dedicated apps.

[0135] User

[0136] Users operate the system and fulfill the following roles:

[0137] 1. Entering meeting schedules and agendas: Enter the meeting date, time, and agenda into the system. For example, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[0138] 2. Upload related materials: Upload the minutes of the previous meeting and related materials using drag-and-drop or file selection dialogs.

[0139] 3. Conference start notification: Notify the system that a conference is about to begin.

[0140] Specific examples

[0141] Take the example of a new product release planning meeting:

[0142] 1. Preparation Phase

[0143] The user enters "New product launch meeting to be held in conference room C at 10:00 on October 1, 2023" into the web form and sets up the meeting schedule in the system.

[0144] Users upload documents such as "minutes of the previous meeting," "specifications for a new product," and "marketing plans" into the system.

[0145] The server stores this information in a database.

[0146] 2. Before the meeting begins

[0147] The server retrieves the meeting agenda and related materials from the database and inputs them into the generative AI model.

[0148] The generative AI model generates key questions and supplementary information based on the prompt sentence and stores it on the server.

[0149] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generative AI model.

[0150] 3. During a meeting

[0151] When a meeting begins, the user tells the system to "start the meeting."

[0152] The device converts the meeting audio into text in real time and sends it to a generative AI model.

[0153] The server uses a generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information.

[0154] The device provides real-time translation, allowing all participants, even those speaking different languages, to understand the content of the meeting.

[0155] 4. After the meeting

[0156] The server creates meeting minutes based on the voice and text data recorded by the generative AI model. For example, it might record, "At the meeting held on October 1, 2023, the release date for the new product was decided to be January 15, 2024."

[0157] The generative AI model summarizes action items for the next meeting based on the discussion, listing specific tasks such as "finish marketing materials by the next meeting."

[0158] The server notifies the user of the created minutes and next action items and stores them in a database.

[0159] In this way, the form for implementing the invention improves the quality and productivity of meetings by coordinating the server, terminals, and users to consistently improve the efficiency of meeting preparation, progress, and follow-up after the meeting has ended.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Step 1:

[0162] Users enter meeting schedules and agendas

[0163] Input: The user enters the date, time, location, and agenda of the meeting into a web form or dedicated app. For example, they might enter, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[0164] Data processing: The input information is sent to the server as text data. The meeting ID, date, time, location, and agenda are parsed and formatted as structured data.

[0165] Output: The formatted data is stored in an internal database.

[0166] Specific behavior: A front-end component (e.g., HTML form, JavaScript) is executed to receive form input and a POST request is executed to send the input to the server.

[0167] Step 2:

[0168] Users upload previous meeting minutes and related materials

[0169] Input: Users upload files such as minutes from a previous meeting, specifications for a new product, or a marketing plan to the system using drag-and-drop operations or file selection dialogs.

[0170] Data processing: Uploaded files are received by the server and stored in a location appropriate for the file format. Metadata (file name, upload date, etc.) is also generated.

[0171] Output: The file body and metadata are saved to the database and file storage.

[0172] Specific behavior: A file selection dialog on the front end, a file upload API endpoint, and a storage system for saving files (e.g. AWS S3, Google Cloud Storage) are used.

[0173] Step 3:

[0174] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database.

[0175] Input: Before a meeting starts, the system queries the database based on the meeting ID entered by the user.

[0176] Data processing: The necessary data (agenda, minutes, related materials) is retrieved from the database and formatted to be input into the generative AI model.

[0177] Output: The formatted data is temporarily stored in memory.

[0178] Specific operations: Executes SQL queries, formats and transforms data, and caches retrieved data in memory.

[0179] Step 4:

[0180] Input the previous meeting minutes and agenda into the generative AI model to generate questions and supplemental information

[0181] Input: Formatted minutes and agenda are sent to the generative AI model with the prompt, "Generate key questions and supporting information to discuss at the next meeting."

[0182] Data Computation: Generative AI models analyze text data and generate questions and follow-up information based on specified prompts.

[0183] Output: The generated questions and supplementary information are returned in text format to the server and stored in an appropriate database.

[0184] Specific operation: An API request is sent to the generative AI model, and the generated content is saved.

[0185] Step 5:

[0186] User tells the system to "start a meeting"

[0187] Input: To start a meeting, the user clicks the "Start Meeting" button on the dedicated app or web interface.

[0188] Data processing: The server notifies the entire system that the conference has started and performs the necessary initialization processing.

[0189] Output: The conference state is updated to "Started" and relevant system components are notified.

[0190] Specific operation: The button click event is processed, the conference state management system is updated, and each component is notified.

[0191] Step 6:

[0192] The device recognizes the audio of the meeting in real time and converts it into text.

[0193] Input: Voice data collected by the device's microphone during the meeting is sent to the voice recognition system.

[0194] Data processing: A speech recognition system (such as Google Cloud Speech-to-Text or Azure Speech Services) analyzes the voice data and converts it into text data in real time.

[0195] Output: The text of the remarks is sent to the server and saved as a meeting record.

[0196] Specific operations: Collects voice data, calls voice recognition API, and sends and saves text data.

[0197] Step 7:

[0198] Generative AI models monitor meeting progress in real time and provide relevant questions and follow-up information

[0199] Input: Real-time transcribed meeting content is fed into the generative AI model.

[0200] Data Computation: The generative AI model analyzes the supplied text data and generates appropriate questions and supplemental information.

[0201] Output: Generated questions and supplemental information are provided to meeting participants in real time.

[0202] Specific operation: Real-time data is supplied to the generative AI model, and the generated content is displayed on the interface.

[0203] Step 8:

[0204] The device performs real-time translation

[0205] Input: Text data output from the generative AI model and speech recognition system is sent to the real-time translation system.

[0206] Data calculation: A translation system (Google Translate API or DeepL API) translates the text data into the specified language.

[0207] Output: The translated text is provided to meeting participants in real time.

[0208] Specific operation: Text data is sent to the translation API and the translated text is displayed.

[0209] Step 9:

[0210] After the meeting, the server creates minutes based on the generative AI model and summarizes next action items.

[0211] Input: Recorded audio and text data is sent to the generative AI model.

[0212] Data Computation: Generative AI models analyze text data and summarize meeting minutes and next action items.

[0213] Output: The generated minutes and next action items are stored in a database.

[0214] Specific operations: Data is sent to the generative AI model, and meeting minutes and action items are automatically generated and saved.

[0215] Step 10:

[0216] The server notifies you of the minutes created and a summary of upcoming action items

[0217] Input: The generated meeting minutes and action items are retrieved from the database.

[0218] Data Processing: Notification content is formatted into a user-friendly format.

[0219] Output: The formatted notification content is sent to the user via email and / or in-app notification.

[0220] Specific operations: Retrieving data from the database, formatting the notification content, and sending the notification.

[0221] (Application example 1)

[0222] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0223] There is a demand for a system that can centrally manage meetings and briefings held within factories, including efficient progress, information sharing, and multilingual support. In addition, collaboration between robots and humans can improve productivity, and real-time speech recognition and translation, as well as the provision of appropriate questions and supplementary information, are necessary. However, conventional systems have difficulty in effectively performing these tasks, consuming a great deal of time and resources. This invention was devised to solve these problems and realize efficient and effective meeting progress.

[0224] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0225] In this invention, the server includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for saving the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting the meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; means for using the saved data to prepare for the next meeting; means for streamlining meetings and briefings held in factories and supporting collaboration between humans and robots; and means for converting the meeting content into text in real time, providing translations, and for the generation AI to provide appropriate questions and supplementary information. This enables meetings to be conducted efficiently and effectively in factories, improving productivity through collaboration between robots and humans.

[0226] "Means for inputting meeting schedules and agendas" refers to a function for registering meeting dates and topics in the system in advance.

[0227] "Means for receiving minutes of previous meetings and related materials" is a function for uploading records of past meetings and necessary materials to the system.

[0228] "Means for storing received information" refers to a function for storing uploaded minutes and materials in a database.

[0229] "Means of inputting the previous minutes and agenda into the generating AI and generating questions and supplementary information" refers to a function that inputs saved minutes and agendas into the generating AI model and automatically generates questions and supplementary information required for the meeting.

[0230] The "means for notifying the start of the conference" is a function for notifying participants of the start time of the conference.

[0231] "Means of converting meeting content into text in real time using voice recognition technology" is a function that uses technology to convert statements made during a meeting into text data in real time.

[0232] "Means for the generative AI to monitor the progress of the meeting in real time and provide appropriate questions or supplementary information" refers to a function in which the generative AI monitors the progress of the meeting and instantly provides questions or supplementary information.

[0233] "Means for providing real-time translation of meeting content using generative AI" is a function that translates meeting content in real time to ensure smooth communication between participants who speak different languages.

[0234] "A means for the generation AI to create minutes after the meeting and summarize the next action items" is a function that automatically creates minutes based on the content of the meeting and summarizes the next action items to be taken.

[0235] "Means for notifying the created minutes and a summary of the next action item" is a function for notifying participants of the created minutes and the next task.

[0236] "Means of using saved data to prepare for the next meeting" is a function that helps prepare for the next meeting by referring to past meeting records and materials.

[0237] "A means to streamline meetings and briefings held within factories and support collaboration between humans and robots" is a function that helps robots participate in meetings and information sharing held within factories so that they can proceed efficiently.

[0238] "A means of converting meeting content into text in real time, providing translation, and having a generation AI provide appropriate questions and supplementary information" refers to a function that converts statements into text data in real time, translates them as needed, and then has a generation AI provide questions and supplementary information.

[0239] The system of the present invention improves the efficiency of meetings and briefings in factories and provides a series of functions to support collaboration between humans and robots. The specific configuration and operation of the system are described below.

[0240] System Overview

[0241] 1. Preparation Phase

[0242] Users can use smartphones, tablets, or even the terminals of robots in factories to enter meeting schedules and specific agendas. For example, they can enter information such as "October 1, 2023, 10:00 AM, in Conference Room C in the factory, to hold a meeting to launch a new product line."

[0243] Users upload previous meeting minutes and related materials (e.g., production reports, safety guidelines) into the system.

[0244] This information is stored in a database by the server.

[0245] 2. Before the meeting begins

[0246] The server retrieves the meeting agenda, previous meeting minutes, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT).

[0247] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[0248] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[0249] 3. During a meeting

[0250] The user tells the system to "start a meeting."

[0251] The terminals (smartphones, factory robots) recognize the audio of the meeting in real time and convert it into text using Google Cloud Speech-to-Text.

[0252] The server monitors the progress of the meeting in real time with a generative AI. For example, when a new software product is discussed, the generative AI asks appropriate questions such as, "Please tell me more about the status of testing of the new feature."

[0253] The device uses the Google Translate API to provide real-time translation in multiple languages, ensuring that participants who speak different languages ​​can understand the content.

[0254] 4. After the meeting

[0255] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it could record that "at the meeting held on October 1, 2023, it was decided that the release date for a new product would be January 15, 2024."

[0256] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[0257] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[0258] Examples of specific examples and prompts

[0259] For example, if we take a new product release planning meeting, the flow would be as follows:

[0260] 1. A user schedules a meeting to launch a new product line in conference room C in the factory at 10:00 AM on October 1, 2023.

[0261] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "safety guidelines" to the system.

[0262] 3. The server acquires this information and inputs it into a generative AI model, which generates questions and supplementary information in advance.

[0263] 4. The conference begins and the user tells the system to "start the conference."

[0264] 5. The device converts the meeting audio into text using Google Cloud Speech-to-Text, and the server provides appropriate questions and follow-up questions using generated AI.

[0265] 6. The device uses the Google Translate API for real-time translation, enabling multilingual support.

[0266] 7. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[0267] 8. The server notifies the user of the minutes and next action items and saves them in the database.

[0268] Prompt Sentence Examples

[0269] User: "We will hold a launch meeting for the new product line in conference room C in the factory on October 1, 2023 at 10:00 AM."

[0270] Server: "Please upload the last meeting minutes, new product specifications, and safety guidelines."

[0271] User: "I uploaded it."

[0272] Server: "Do you want to start the meeting?"

[0273] User: "Yes, let's start."

[0274] Server (generative AI): "Tell me more about the challenges you're facing with your current product line."

[0275] Server (generative AI): "Please tell me the status of the commissioning of the new product line."

[0276] Server: "The meeting has ended. The minutes are being compiled..."

[0277] Server: "I've summarized our next action items: 1. Report test results for new product line. 2. Review safety guidelines."

[0278] In this way, a system is provided that enables meetings within a factory to be conducted efficiently and enables collaboration between robots and humans.

[0279] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0280] Step 1:

[0281] Users use smartphones, tablets, or robotic terminals in factories to enter meeting schedules and agendas. The entered data includes the date, time, location, and topic of the meeting. The entered information is sent to a server and stored in a database. The meeting is then prepared.

[0282] Input: Meeting schedule and agenda

[0283] Output: Meeting schedules and agendas stored in a database

[0284] Step 2:

[0285] Users upload minutes of previous meetings and related materials, such as production reports and safety guidelines, to the system, which the server then stores in a database.

[0286] Input: Minutes of previous meetings and related materials

[0287] Output: Meeting minutes and related materials stored in a database

[0288] Step 3:

[0289] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT). The generative AI automatically generates important questions and supplementary information from the meeting content and saves the output on the server.

[0290] Input: Agendas, minutes, and related materials retrieved from the database

[0291] Output: Auto-generated questions and additional information

[0292] Step 4:

[0293] When a conference starts, the user notifies the system that the conference is about to begin. The server then notifies all participants that the conference has begun.

[0294] Input: Meeting start notification

[0295] Output: Notification to participants

[0296] Step 5:

[0297] Devices (smartphones, factory robots) recognize speech during meetings in real time and convert it into text using Google Cloud Speech-to-Text. The converted speech is sent to a server, where a generating AI monitors the progress of the meeting in real time.

[0298] Input: Audio data during the meeting

[0299] Output: Real-time textualized speech data

[0300] Step 6:

[0301] The server analyzes the text of the meeting, which has been converted by the generative AI, and provides appropriate questions and supplementary information. For example, if a new product software is being discussed, the generative AI might ask, "Please tell me more about the status of testing of the new function."

[0302] Input: Translated meeting contents

[0303] Output: Questions and supplementary information generated by the generative AI

[0304] Step 7:

[0305] The device uses the Google Translate API to translate meeting content in real time and display it in a format that is easy to understand for participants who speak different languages, thereby enabling multilingual support.

[0306] Input: Translated meeting contents

[0307] Output: Real-time translated meeting content

[0308] Step 8:

[0309] After the meeting, the server uses generative AI to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0310] Input: Meeting audio and text data

[0311] Output: Generated meeting minutes

[0312] Step 9:

[0313] Based on the discussion, the generative AI summarizes action items for the next meeting, such as listing specific tasks like "complete marketing materials by the next meeting." This provides a concrete guide to action.

[0314] Input: Minutes and meeting contents

[0315] Output: List of next action items

[0316] Step 10:

[0317] The server notifies the user of the created minutes and next action items and stores them in a database, allowing the user to check the minutes and next action items, thereby streamlining preparations for the next meeting.

[0318] Input: Generated meeting minutes and action items

[0319] Output: Notify user and save to database

[0320] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0321] The system of this invention provides a series of functions for enhancing conferences through collaboration between servers, terminals, and users. Furthermore, by combining it with an emotion engine that recognizes users' emotions, more effective conference management is achieved. Below, we explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[0322] overview

[0323] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for asking questions and providing additional information, real-time translation, creating meeting minutes, summarizing upcoming action items, and recognizing user emotions with an emotion engine.

[0324] System execution example

[0325] 1. Preparation Phase

[0326] The user inputs the meeting schedule and specific agenda into the system. For example, the user might input "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0327] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[0328] This information is stored in a database by the server.

[0329] 2. Before the meeting begins

[0330] The server retrieves the saved meeting agenda, previous minutes, and related materials from the database and inputs this information into the generative AI model.

[0331] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[0332] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[0333] 3. During a meeting

[0334] When a meeting begins, the user tells the system to "start the meeting."

[0335] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[0336] The server monitors the progress of the meeting in real time through generative AI and asks appropriate questions, such as, "Please tell me the status of software testing for the new product."

[0337] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions.

[0338] The recognized emotion data is sent to a server, and the generation AI adjusts the questions and supplementary information based on that information. For example, if the user appears confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[0339] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[0340] 4. After the meeting

[0341] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the release date for the new product will be January 15, 2024."

[0342] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[0343] Analyzes emotional data after the meeting and generates a report evaluating satisfaction and responsiveness of the meeting.

[0344] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[0345] Specific examples

[0346] For example, if we take a new product release planning meeting, the flow would be as follows:

[0347] 1. A user schedules a new product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00.

[0348] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "marketing plan" to the system.

[0349] 3. The server obtains this information and inputs it into a generative AI model to generate questions and supplementary information in advance.

[0350] 4. The conference begins and the user tells the system to "start the conference."

[0351] 5. The device converts the audio of the meeting into text, and the server provides appropriate questions and follow-up questions using generated AI.

[0352] 6. The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions, and the data is sent to the server.

[0353] 7. The server uses a generative AI to adjust questions and follow-up questions based on emotional data, and if the user appears confused, it will ask, "Could you please explain in more detail?"

[0354] 8. The device will perform real-time translation, enabling multilingual support.

[0355] 9. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[0356] 10. Generative AI analyzes emotional data during meetings and generates reports evaluating satisfaction and responsiveness of meetings.

[0357] 11. The server notifies the user of the minutes and next action items and saves them in the database.

[0358] In this way, the AI-Meeting Assistant system, which combines an emotion engine, provides comprehensive support from meeting preparation to progress and post-conclusion follow-up, helping to ensure fruitful meetings. By utilizing emotion data, it becomes possible to respond appropriately to the user's reactions, leading to more effective meeting management.

[0359] The processing flow will be explained below.

[0360] Step 1:

[0361] A user logs in to the system and enters the schedule and agenda for a meeting. For example, the user enters, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0362] Step 2:

[0363] A user uploads the minutes of a previous meeting and related documents to the system, such as a specification document for a new product or a marketing plan.

[0364] Step 3:

[0365] The server receives the meeting schedule, agenda, and related materials sent by the user, and then categorizes and stores the information in a database.

[0366] Step 4:

[0367] The server retrieves the saved meeting agenda, previous meeting minutes, and related materials from the database and inputs them into the generative AI model.

[0368] Step 5:

[0369] The generative AI analyzes the meeting content and generates important questions and supplementary information in advance, which is then stored on the server.

[0370] Step 6:

[0371] The server assigns roles to the generation AI, such as "questioner," "supplementary information provider," and "minutes creator."

[0372] Step 7:

[0373] When it is time for the meeting to start, the user notifies the system that the meeting should begin, for example by clicking a button.

[0374] Step 8:

[0375] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[0376] Step 9:

[0377] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions, for example, determining whether they are confused or excited.

[0378] Step 10:

[0379] The emotion engine sends the analyzed emotion data to the server, which then passes this data to the generation AI.

[0380] Step 11:

[0381] The server monitors the progress of the meeting in real time using a generative AI, which then asks appropriate questions to participants, such as, "Please tell me about the status of software testing for the new product."

[0382] Step 12:

[0383] The generative AI adjusts the user's response based on emotional data, providing further explanations or asking follow-up questions if the user is confused or does not understand.

[0384] Step 13:

[0385] Users can answer questions posed by the AI, further deepening the discussion, and the AI ​​can provide additional information as needed.

[0386] Step 14:

[0387] The device translates the meeting content in real time, allowing participants who speak different languages ​​to understand the content.

[0388] Step 15:

[0389] After the meeting, the server uses the AI ​​to create minutes based on the audio and text data of the meeting. For example, the minutes could read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0390] Step 16:

[0391] The generative AI summarizes the next action items discussed during the meeting, listing specific tasks such as "finish marketing materials by the next meeting."

[0392] Step 17:

[0393] The server notifies the user of the created minutes and a summary of the next action items, for example, via email or in-system notifications.

[0394] Step 18:

[0395] The emotion engine analyzes post-meeting emotional data and generates a report assessing meeting satisfaction and responsiveness.

[0396] Step 19:

[0397] Use the meeting information, minutes, and summary of next action items stored in the database to prepare for the next meeting, making it more effective.

[0398] Example 2

[0399] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0400] Conventional meeting support systems required the use of multiple tools for meeting preparation, progress, and follow-up, making efficient meeting management difficult. In particular, they were unable to grasp participants' emotions and reactions in real time and provide appropriate questions or supplementary information based on that information. Communication between participants who spoke different languages ​​also posed an obstacle to meeting management.

[0401] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0402] In this invention, the server includes means for inputting meeting schedules and agendas, means for receiving minutes and related materials from the previous meeting, means for saving the received information, means for inputting the minutes and agenda from the previous meeting to a generation AI and generating questions and supplementary information, means for notifying the start of the meeting, means for converting the meeting content into text in real time using speech recognition technology, means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information, means for the generation AI to provide real-time translation of the meeting content, means for recognizing user emotions using an emotion engine and for the generation AI to adjust the questions and supplementary information based on the recognized emotion data, means for the generation AI to create minutes and summarize next action items after the meeting ends, means for analyzing emotion data and generating a report evaluating satisfaction and responsiveness, means for notifying the created minutes and summaries of next action items, and means for using the saved data to prepare for the next meeting. This provides consistent support for meetings from preparation to progress and follow-up after the end, enabling appropriate responses based on user emotions, thereby improving the efficiency and quality of meetings.

[0403] "Scheduled Meeting" means a specific schedule of a meeting to be held on a specific date and time.

[0404] "Agenda" means the list of items or topics to be discussed at a meeting.

[0405] "Meeting minutes" refers to a document that records what was discussed and what decisions were made during a meeting.

[0406] "Related Materials" means any additional information or documents related to the agenda of the meeting.

[0407] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate questions and supplemental information.

[0408] "Voice recognition technology" refers to technology that converts voice into text data.

[0409] An "emotion engine" refers to a system that recognizes emotions by analyzing a user's facial expressions and tone of voice.

[0410] "Real-time translation" refers to technology that translates conversations into different languages ​​in real time.

[0411] "Meeting proceedings" refers to the procedures and processes for smoothly running a meeting.

[0412] "Next action items" refer to specific tasks or action items to be carried out before the next meeting.

[0413] "Emotional data" refers to data that indicates the emotional state of a user.

[0414] "Evaluation Report" means a report that evaluates the satisfaction and responsiveness of a meeting based on specific criteria.

[0415] The system of the present invention provides a series of functions that consistently support the preparation, progress, and follow-up of a conference. The system operates in cooperation with a server, terminals, and users. Each function of the system is implemented as follows:

[0416] 1. Schedule meetings and enter agendas

[0417] Users input meeting schedules and agendas into the system. This information is stored in a database by the server. For example, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0418] 2. Uploading materials

[0419] Users upload the minutes of previous meetings and related documents (e.g., specifications for new products, marketing plans) to the system, and the server stores this information in a database.

[0420] 3. Input of meeting materials

[0421] The server retrieves the meeting agenda and related materials from the database and inputs them into the generative AI model. The generative AI model analyzes this data and generates important questions and supplemental information from the meeting content. For example, it generates a question such as, "Please tell me the status of software testing for the new product."

[0422] 4. Role assignment

[0423] The server assigns different roles to the generating AI, such as questioner, supplementary information provider, and minutes writer.

[0424] 5. Before the meeting begins

[0425] When a meeting starts, the user notifies the system that they want to start a meeting. The server then passes the prepared data to the generation AI, which prepares to support the progress of the meeting.

[0426] 6. Speech Recognition and Text Conversion

[0427] The device recognizes the audio of the meeting in real time and converts it into text format. This process records all utterances as text. For example, a utterance such as "When is the release date of the new product?" is sent to the server as text data.

[0428] 7. Real-time monitoring and questioning

[0429] The server uses generative AI to monitor the progress of the meeting in real time and provide appropriate questions and supplemental information. Based on the content of the meeting, the generative AI asks questions such as, "Please tell me about the status of software testing for our new product."

[0430] 8. Analysis by Emotion Engine

[0431] The device has a built-in emotion engine that analyzes the user's facial expressions and tone of voice to recognize their emotions. The recognized emotion data is sent to a server, and the generation AI adjusts the questions and supplementary information based on that data. For example, if the user is confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[0432] 9. Real-time translation

[0433] The device translates in real time during a meeting, ensuring that participants who speak different languages ​​can understand what is being said, for example translating what is being said in English into Japanese.

[0434] 10. Preparation of minutes

[0435] After the meeting, the server uses generative AI to create minutes based on the audio and text data from the meeting. For example, it might record, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0436] 11. Summary of next action items

[0437] The generative AI model then summarizes the next action items based on the discussion, such as a list of specific tasks such as "finish marketing materials by the next meeting."

[0438] 12. Emotion Data Analysis and Report Generation

[0439] After the meeting, the server analyzes the emotional data and generates a report assessing satisfaction and responsiveness of the meeting.

[0440] 13. Notice and Retention

[0441] The created minutes and next action items are notified to the user by the server and stored in a database.

[0442] This allows the system to provide consistent support from meeting preparation to progress and follow-up after the meeting has ended, and enables appropriate responses based on the user's emotions, thereby improving the efficiency and quality of meetings.

[0443] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0444] Step 1:

[0445] The user enters the meeting schedule and agenda. The "meeting date and time," "location," and "topic" entered by the user are sent to the system. The server saves this information in a database. This allows the system to manage basic meeting information. Example input: "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[0446] Step 2:

[0447] Users upload the minutes of previous meetings and related materials to the system. Related materials provided by users, such as "minutes," "specifications," and "marketing plans," are sent to the system. The server receives these materials and stores them in a database. This provides background information for the meeting.

[0448] Step 3:

[0449] The server retrieves the saved meeting agenda and related materials from the database. The retrieved "agenda," "minutes," and "materials" are input into the generative AI model. The generative AI model analyzes this data and generates important questions and supplementary information from the meeting content. The generated questions are saved on the server. For example, it generates a question such as, "Please tell me the status of software testing for the new product."

[0450] Step 4:

[0451] The server assigns roles to the generating AI, such as "questioner," "supplementary information provider," and "minutes writer," which determines the role the generating AI will play during the meeting.

[0452] Step 5:

[0453] When a meeting starts, the user notifies the system that they want to start a meeting. The server receives this notification and the entire system switches to meeting mode. The server then supplies pre-prepared data to the generation AI and begins supporting the progress of the meeting.

[0454] Step 6:

[0455] The device recognizes the audio of the meeting in real time and converts it into text. The input audio data is converted into "text data" using the device's voice recognition technology. The converted text data is sent to the server and stored. For example, the audio "When is the release date for the new product?" is converted into text.

[0456] Step 7:

[0457] The server uses generative AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information. It analyzes what is being said in the meeting in real time and asks questions based on the content of the meeting. For example, it might ask, "Please tell me about the status of software testing for our new product."

[0458] Step 8:

[0459] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions. The analyzed "emotion data" is sent from the device to a server. The server passes the received emotion data to the generation AI, which then adjusts the questions and supplementary information based on the emotion data. For example, if the user seems confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[0460] Step 9:

[0461] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand the content. Based on the input "voice data" and "text data," "translated text" is generated and provided to other participants. For example, content spoken in English can be translated into Japanese.

[0462] Step 10:

[0463] After the meeting ends, the server uses a generation AI to create minutes based on the audio and text data from the meeting. The generation AI analyzes the input audio and text data and generates minutes text. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the release date for the new product will be January 15, 2024."

[0464] Step 11:

[0465] The generative AI model summarizes next action items based on the content of the discussion. It analyzes the "minutes text" as input and generates "next action items." For example, it lists specific tasks such as "Complete marketing materials by the next meeting."

[0466] Step 12:

[0467] After the meeting ends, the server analyzes the emotional data and generates a report evaluating the satisfaction and responsiveness of the meeting. Based on the input "emotional data," a "satisfaction evaluation report" and a "responsiveness evaluation report" are generated. This allows the quality of the meeting to be objectively evaluated.

[0468] Step 13:

[0469] The server notifies the user of the minutes and next action items, and stores them in a database, which can be used to prepare for the next meeting.

[0470] (Application example 2)

[0471] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0472] Current meeting management systems often require manual preparation, progress, and follow-up after a meeting, which is labor-intensive and time-consuming. They also face challenges in being unable to meet advanced requirements, such as conducting meetings in multiple languages ​​or recognizing participants' emotions. In particular, there is a lack of systems that can properly manage workers' emotions and stress in production line work environments. Therefore, there is a need for a system that can simultaneously improve meeting efficiency and consider the emotions of workers.

[0473] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0474] In this invention, the server includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for saving the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting the meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; an emotion engine that recognizes participants' emotions during the meeting and analyzes the emotion data to adjust questions and supplementary information; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; and means for using the saved data to prepare for the next meeting. This provides comprehensive support for meetings from preparation to progress and follow-up after the meeting, and by utilizing emotion data, it is possible to appropriately manage worker reactions and stress, enabling effective meeting management and improving the working environment.

[0475] The "means for inputting meeting schedules and agendas" is a function that allows a user to input schedule information such as the date and time of a meeting and the agenda into the system.

[0476] "Means for receiving minutes of the previous meeting and related materials" is a function for uploading minutes created at the previous meeting and materials related to the meeting to the system.

[0477] "Means for storing received information" refers to a function for storing meeting information and materials entered or uploaded into the system in a database.

[0478] "Means of inputting the previous meeting minutes and agenda into the generating AI and generating questions and supplementary information" refers to a function that inputs the previous meeting minutes and agenda into the AI ​​and automatically generates important questions and supplementary information based on them.

[0479] The "means for notifying the start of a conference" is a function for notifying the system when a user starts a conference.

[0480] "Means for converting meeting content into text in real time using voice recognition technology" is a function that uses voice recognition technology to convert voices spoken during a meeting into text in real time.

[0481] "Means for generative AI to monitor the progress of the meeting in real time and provide appropriate questions or supplementary information" refers to a function that monitors the progress of the meeting in real time and allows the AI ​​to provide questions or supplementary information at the appropriate time.

[0482] "Means for providing real-time translation of meeting content using generative AI" refers to a function that uses AI to translate meeting content in real time so that meeting participants who speak different languages ​​can understand the content.

[0483] "Means for adjusting questions and supplementary information by analyzing emotional data, including an emotion engine that recognizes participants' emotions during the meeting" refers to a function that uses an emotion engine that analyzes participants' facial expressions and tone of voice to obtain emotional data during the meeting, and then uses the AI ​​to adjust questions and supplementary information based on that data.

[0484] "A means for the generation AI to create minutes after the meeting and summarize action items for the next meeting" is a function in which the AI ​​automatically creates minutes based on the meeting content after the meeting and summarizes action items for the next meeting.

[0485] The "means for notifying the user of the created minutes and a summary of the next action item" is a function for notifying the user of the created minutes and a summary of the next action item.

[0486] "Means for using saved data in preparation for the next meeting" is a function for utilizing past meeting data saved in the database in preparation for the next meeting.

[0487] The system for implementing this invention is configured to provide various functions through cooperation between a server, a terminal, and a user. This system includes programs for realizing the following functions based on the claims.

[0488] First, the user enters the meeting schedule and agenda, and uploads the minutes of the previous meeting and related materials to the system. This information is sent to the server via the device and stored in the server's database. The server then inputs the stored information into the generation AI, which automatically generates questions and supplementary information based on the meeting content.

[0489] At the start of a meeting, the user uses their device to announce "start meeting." The device then converts the meeting content into text in real time using voice recognition technology, and the generation AI provides real-time questions and supplementary information. Furthermore, during the meeting, an emotion engine recognizes the emotions of participants and sends this data to the server. The generation AI then adjusts the questions and supplementary information based on the emotion data, optimizing the progress of the meeting. The generation AI also provides real-time translation for multilingual meeting participants.

[0490] After the meeting ends, the server uses generative AI to create minutes based on the audio and text data of the meeting and summarizes the next action items. This information is notified to the user and saved in a database. The saved data can be used to prepare for the next meeting.

[0491] As a specific example, let's imagine a new product release planning meeting, and the flow would be as follows:

[0492] 1. A user schedules a new product launch meeting to be held in the conference room on October 1, 2023 at 10:00 AM.

[0493] 2. Upload previous meeting minutes, new product specifications, and marketing plans into the system.

[0494] 3. The server obtains this information and inputs it into a generative AI model to generate questions and supplementary information in advance.

[0495] 4. When the conference starts, the user tells the system to "start the conference."

[0496] 5. The device converts the audio of the meeting into text, and the generative AI provides appropriate questions and follow-up comments.

[0497] 6. The device's built-in emotion engine analyzes the user's facial expressions and tone of voice and sends the emotion data to the server.

[0498] 7. Based on the emotion data, the server will ask questions and provide further explanations. If the user is confused, the AI ​​will ask questions such as, "Could you explain more specifically?"

[0499] 8. The device will translate the meeting content in real time, enabling multilingual support.

[0500] 9. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[0501] 10. The generated minutes and next action items are notified to the user and saved in the database.

[0502] The hardware used includes smartphones and head-mounted displays, and the software used includes the speech_recognition library (speech recognition) and the transformers library (generative AI and emotion recognition).

[0503] Example prompt sentence:

[0504] "A system that recognizes speech and analyzes emotions in real time during new product launch meetings"

[0505] "I want to automatically generate questions based on the presentation content and save them as meeting minutes."

[0506] In this way, the system improves meeting efficiency and the working environment.

[0507] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0508] Step 1:

[0509] The user inputs the meeting schedule and agenda. At this stage, the user inputs information such as "The new product launch meeting will be held on October 1, 2023 at 10:00" through the system interface. This obtains the meeting date and agenda as input data.

[0510] Step 2:

[0511] The user uploads the minutes of the previous meeting and related materials to the system. In this step, the user sends the minutes of the previous meeting, the specifications of the new product, the marketing plan, and other related materials to the system, which then receives these materials as input data.

[0512] Step 3:

[0513] The server stores the received information. The server accumulates the meeting schedule, agenda, and related materials obtained in steps 1 and 2 in a database. This stores the received information in the database and saves the accumulated data.

[0514] Step 4:

[0515] The server inputs the previous meeting minutes and agenda into the generation AI, which then generates questions and supplementary information. The server inputs the previous meeting minutes and agenda saved from the database into the generation AI model. The generation AI analyzes this data and automatically generates important questions and supplementary information related to the meeting content. The generated questions and supplementary information are obtained as output data.

[0516] Step 5:

[0517] The user notifies the system of the start of the conference. The user inputs "start conference" into the system via the terminal, and the conference starts. This acquires the conference start notification as input data.

[0518] Step 6:

[0519] The device uses speech recognition technology to convert the contents of the meeting into text in real time. The device picks up the meeting audio with its built-in microphone and converts the audio into text using the speech_recognition library. At this stage, the real-time converted text becomes the output data.

[0520] Step 7:

[0521] The server uses generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information. The server monitors the text data of the meeting content generated in step 6 and adds automatically generated questions and supplementary information using generation AI at the appropriate time. As a result, the questions and supplementary information provided become the output.

[0522] Step 8:

[0523] The device uses AI generation to provide real-time translation of the meeting content. The device translates text data in real time to support multiple languages, allowing participants who speak multiple languages ​​to understand the content. At this stage, the real-time translated meeting content is obtained as the output.

[0524] Step 9:

[0525] The device uses an emotion engine to recognize participants' emotions during the meeting and analyzes the emotion data to adjust questions and follow-up information. The device's built-in camera and microphone analyze facial expressions and vocal tone to recognize emotions. Based on this analysis, the emotion data is sent to a server, which then uses generative AI to adjust questions and follow-up information. This allows questions and follow-up information to be output that are adapted to the emotions.

[0526] Step 10:

[0527] The server uses the generation AI to create meeting minutes and summarize action items for the next meeting. After the meeting ends, the server uses the meeting content text data from step 6 and the emotion data from step 9 to automatically generate meeting minutes using the generation AI. In addition, based on the matters discussed during the meeting, the server summarizes action items for the next meeting. This outputs specific action items for the next meeting.

[0528] Step 11:

[0529] The server notifies the user of the created minutes and next action items. The server notifies the user of the automatically generated minutes and next action items and also stores them in a database. The final minutes and action items are notified to the user and are also stored as preparation data for the next meeting.

[0530] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0531] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0532] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0533] [Second embodiment]

[0534] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0535] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0536] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0537] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0538] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0539] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0540] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0541] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0542] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0543] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0544] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0545] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0546] The system of this invention provides a series of functions for enhancing conferences through collaboration between the server, terminals, and users. Below, we will explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[0547] overview

[0548] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for questions and supplementary information, real-time translation, meeting minutes, and a summary of upcoming action items.

[0549] System execution example

[0550] 1. Preparation Phase

[0551] The user inputs the meeting schedule and specific agenda into the system. For example, the user might input "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0552] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[0553] This information is stored in a database by the server.

[0554] 2. Before the meeting begins

[0555] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into the generative AI model.

[0556] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[0557] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[0558] 3. During a meeting

[0559] When a meeting begins, the user tells the system to "start the meeting."

[0560] The device recognizes the audio of the meeting in real time and converts it into text.

[0561] The server monitors the progress of the meeting in real time with a generative AI. For example, if a new product software is being discussed, the generative AI will ask appropriate questions, such as, "Could you please tell me more about the status of testing of the new feature?"

[0562] The user's statements are recorded, and the generating AI provides supplementary information as needed.

[0563] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[0564] 4. After the meeting

[0565] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0566] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[0567] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[0568] Specific examples

[0569] For example, if we take a new product release planning meeting, the flow would be as follows:

[0570] 1. A user schedules a new product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00.

[0571] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "marketing plan" to the system.

[0572] 3. The server obtains this information and inputs it into the generative AI model, which generates questions and supplementary information in advance.

[0573] 4. The conference begins and the user notifies the system to "start the conference."

[0574] 5. The device converts the audio of the meeting into text, and the server uses generated AI to provide appropriate questions and additional information.

[0575] 6. The device performs real-time translation, enabling multilingual support.

[0576] 7. After the meeting, the server uses AI to create minutes and summarize next action items.

[0577] 8. The server notifies the user of the minutes and next action items and stores them in the database.

[0578] In this way, the AI-Meeting Assistant system provides comprehensive support from meeting preparation to progress and follow-up after the meeting has ended, helping to ensure fruitful meetings.

[0579] The processing flow will be explained below.

[0580] Step 1:

[0581] A user logs in to the system and enters the schedule and agenda for a meeting. For example, the user enters, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0582] Step 2:

[0583] A user uploads the minutes of a previous meeting and related documents to the system, such as a specification document for a new product or a marketing plan.

[0584] Step 3:

[0585] The server receives the meeting schedule, agenda, and related materials sent by the user, and then categorizes and stores the information in a database.

[0586] Step 4:

[0587] The server retrieves the saved meeting agenda, previous meeting minutes, and related materials from the database and inputs them into the generative AI model.

[0588] Step 5:

[0589] The generative AI analyzes the meeting content and generates important questions and supplementary information in advance, which is then stored on the server.

[0590] Step 6:

[0591] The server assigns roles to the generation AI, such as "questioner," "supplementary information provider," and "minutes creator."

[0592] Step 7:

[0593] When it is time for the meeting to start, the user notifies the system that the meeting should begin, for example by clicking a button.

[0594] Step 8:

[0595] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[0596] Step 9:

[0597] The server monitors the progress of the meeting in real time through a generative AI, which then asks appropriate questions to participants, such as, "Please tell me about the status of testing the software for our new product."

[0598] Step 10:

[0599] Users can answer questions posed by the AI, further deepening the discussion, and the AI ​​can provide additional information as needed.

[0600] Step 11:

[0601] The device translates the meeting content in real time, allowing participants who speak different languages ​​to understand the content.

[0602] Step 12:

[0603] After the meeting, the server uses the AI ​​to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0604] Step 13:

[0605] The generative AI summarizes the next action items discussed during the meeting, listing specific tasks such as "finish marketing materials by the next meeting."

[0606] Step 14:

[0607] The server notifies the user of the created minutes and a summary of the next action items, for example, via email or in-system notifications.

[0608] Step 15:

[0609] Use the meeting information, minutes, and summary of next action items stored in the database to prepare for the next meeting, making it more effective.

[0610] Example 1

[0611] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0612] In conventional conferencing systems, meeting preparation, proceeding, and minutes creation are done manually, which requires time and effort. Furthermore, difficulties with real-time speech recognition and translation, and the provision of appropriate questions and supplementary information, result in a decline in meeting efficiency and quality. This invention aims to solve these problems by automating meeting preparation, proceeding, minutes creation, and summarizing next action items.

[0613] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0614] In this invention, the server includes means for inputting meeting schedules and agendas, means for receiving minutes and related materials from the previous meeting, means for storing the received information, means for inputting the minutes and agenda from the previous meeting into a generative AI model and generating questions and supplementary information, means for notifying the start of the meeting, means for converting the meeting content into text in real time using speech recognition technology, means for the generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information, means for the generative AI model to provide real-time translation of the meeting content, means for the generative AI model to create minutes and summarize next action items after the meeting ends, means for notifying the created minutes and summaries of next action items, and means for using the stored data to prepare for the next meeting. This makes it possible to streamline a series of meeting processes, such as preparing for the meeting, conducting the meeting, creating minutes, and summarizing next action items, thereby improving the quality and productivity of meetings.

[0615] A "meeting schedule and agenda entry means" is an interface through which a user enters meeting dates, times, locations, and agendas into the system.

[0616] The "means for receiving minutes of previous meetings and related materials" is an interface that allows users to upload records of previous meetings and documents related to meetings to the system.

[0617] The "means for storing received information" is a function for storing the meeting schedule and agenda entered by the user and the uploaded materials in a database.

[0618] "Means of inputting previous meeting minutes and agendas into a generative AI model to generate questions and follow-up information" refers to the process and techniques for inputting stored meeting data into a generative AI model to generate important questions and follow-up information.

[0619] The "means for notifying the start of a conference" is an operation interface that allows a user to notify the system of the start of a conference.

[0620] "Means for converting meeting content into text in real time using speech recognition technology" refers to technology and devices for converting speech made during a meeting from voice to text in real time.

[0621] "Means for a generative AI model to monitor the progress of a meeting in real time and provide appropriate questions and supplementary information" refers to a technology that enables a generative AI model to analyze the content of a meeting, ask appropriate questions on the spot, and provide supplementary information.

[0622] "Means for providing real-time translation of meeting content using generative AI models" refers to technologies and devices that translate speech in real time during a meeting so that all participants who speak different languages ​​can understand the content.

[0623] "Means for a generative AI model to create minutes after a meeting and summarize action items for the next meeting" refers to a technology that analyzes the contents of a meeting from recorded audio and text data, and generates minutes and action items to be addressed at the next meeting.

[0624] The "means for notifying the user of the created minutes and a summary of the next action item" is a function for notifying the user of the created minutes and action items.

[0625] The "means for utilizing stored data in preparation for the next meeting" refers to the process and technology for utilizing stored meeting data in preparation for the next meeting.

[0626] The system of the present invention provides a series of functions for efficiently proceeding and managing a conference through collaboration between a server, terminals, and users. An embodiment of the program of this system will be described in detail below.

[0627] System Configuration

[0628] server

[0629] The server plays a central role in the system and provides the following functions:

[0630] 1. Database management: Store, retrieve, and manage meeting schedules, agendas, minutes, related materials, etc. MySQL or PostgreSQL is used as the RDBMS.

[0631] 2. Generative AI model interface: Input data into a generative AI model (e.g., OpenAI GPT-4) and have it generate questions and supplemental information. The generative AI model operates by receiving the following prompts:

[0632] "Generate important questions and supporting information to discuss at the next meeting."

[0633] 3. Real-time data processing: Audio and text data from meetings are processed in real time and passed to the generative AI model.

[0634] Terminal

[0635] The device is used at the meeting site and provides the following features:

[0636] 1. Speech Recognition: Converts meeting audio into text in real time using Google Cloud Speech-to-Text and Azure Speech Services.

[0637] 2. Real-time translation: Translates meeting content in real time, enabling multilingual support. Translation is performed using the Google Translate API and DeepL API.

[0638] 3. User operation interface: Provides a UI for users to start meetings and upload materials, using web forms or dedicated apps.

[0639] User

[0640] Users operate the system and fulfill the following roles:

[0641] 1. Entering meeting schedules and agendas: Enter the meeting date, time, and agenda into the system. For example, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[0642] 2. Upload related materials: Upload the minutes of the previous meeting and related materials using drag-and-drop or file selection dialogs.

[0643] 3. Conference start notification: Notify the system that a conference is about to begin.

[0644] Specific examples

[0645] Take the example of a new product release planning meeting:

[0646] 1. Preparation Phase

[0647] The user enters "New product launch meeting to be held in conference room C at 10:00 on October 1, 2023" into the web form and sets up the meeting schedule in the system.

[0648] Users upload documents such as "minutes of the previous meeting," "specifications for a new product," and "marketing plans" into the system.

[0649] The server stores this information in a database.

[0650] 2. Before the meeting begins

[0651] The server retrieves the meeting agenda and related materials from the database and inputs them into the generative AI model.

[0652] The generative AI model generates key questions and supplementary information based on the prompt sentence and stores it on the server.

[0653] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generative AI model.

[0654] 3. During a meeting

[0655] When a meeting begins, the user tells the system to "start the meeting."

[0656] The device converts the meeting audio into text in real time and sends it to a generative AI model.

[0657] The server uses a generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information.

[0658] The device provides real-time translation, allowing all participants, even those speaking different languages, to understand the content of the meeting.

[0659] 4. After the meeting

[0660] The server creates meeting minutes based on the voice and text data recorded by the generative AI model. For example, it might record, "At the meeting held on October 1, 2023, the release date for the new product was decided to be January 15, 2024."

[0661] The generative AI model summarizes action items for the next meeting based on the discussion, listing specific tasks such as "finish marketing materials by the next meeting."

[0662] The server notifies the user of the created minutes and next action items and stores them in a database.

[0663] In this way, the form for implementing the invention improves the quality and productivity of meetings by coordinating the server, terminals, and users to consistently improve the efficiency of meeting preparation, progress, and follow-up after the meeting has ended.

[0664] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0665] Step 1:

[0666] Users enter meeting schedules and agendas

[0667] Input: The user enters the date, time, location, and agenda of the meeting into a web form or dedicated app. For example, they might enter, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[0668] Data processing: The input information is sent to the server as text data. The meeting ID, date, time, location, and agenda are parsed and formatted as structured data.

[0669] Output: The formatted data is stored in an internal database.

[0670] Specific behavior: A front-end component (e.g., HTML form, JavaScript) is executed to receive form input and a POST request is executed to send the input to the server.

[0671] Step 2:

[0672] Users upload previous meeting minutes and related materials

[0673] Input: Users upload files such as minutes from a previous meeting, specifications for a new product, or a marketing plan to the system using drag-and-drop operations or file selection dialogs.

[0674] Data processing: Uploaded files are received by the server and stored in a location appropriate for the file format. Metadata (file name, upload date, etc.) is also generated.

[0675] Output: The file body and metadata are saved to the database and file storage.

[0676] Specific behavior: A file selection dialog on the front end, a file upload API endpoint, and a storage system for saving files (e.g. AWS S3, Google Cloud Storage) are used.

[0677] Step 3:

[0678] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database.

[0679] Input: Before a meeting starts, the system queries the database based on the meeting ID entered by the user.

[0680] Data processing: The necessary data (agenda, minutes, related materials) is retrieved from the database and formatted to be input into the generative AI model.

[0681] Output: The formatted data is temporarily stored in memory.

[0682] Specific operations: Executes SQL queries, formats and transforms data, and caches retrieved data in memory.

[0683] Step 4:

[0684] Input the previous meeting minutes and agenda into the generative AI model to generate questions and supplemental information

[0685] Input: Formatted minutes and agenda are sent to the generative AI model with the prompt, "Generate key questions and supporting information to discuss at the next meeting."

[0686] Data Computation: Generative AI models analyze text data and generate questions and follow-up information based on specified prompts.

[0687] Output: The generated questions and supplementary information are returned in text format to the server and stored in an appropriate database.

[0688] Specific operation: An API request is sent to the generative AI model, and the generated content is saved.

[0689] Step 5:

[0690] User tells the system to "start a meeting"

[0691] Input: To start a meeting, the user clicks the "Start Meeting" button on the dedicated app or web interface.

[0692] Data processing: The server notifies the entire system that the conference has started and performs the necessary initialization processing.

[0693] Output: The conference state is updated to "Started" and relevant system components are notified.

[0694] Specific operation: The button click event is processed, the conference state management system is updated, and each component is notified.

[0695] Step 6:

[0696] The device recognizes the audio of the meeting in real time and converts it into text.

[0697] Input: Voice data collected by the device's microphone during the meeting is sent to the voice recognition system.

[0698] Data processing: A speech recognition system (such as Google Cloud Speech-to-Text or Azure Speech Services) analyzes the voice data and converts it into text data in real time.

[0699] Output: The text of the remarks is sent to the server and saved as a meeting record.

[0700] Specific operations: Collects voice data, calls voice recognition API, and sends and saves text data.

[0701] Step 7:

[0702] Generative AI models monitor meeting progress in real time and provide relevant questions and follow-up information

[0703] Input: Real-time transcribed meeting content is fed into the generative AI model.

[0704] Data Computation: The generative AI model analyzes the supplied text data and generates appropriate questions and supplemental information.

[0705] Output: Generated questions and supplemental information are provided to meeting participants in real time.

[0706] Specific operation: Real-time data is supplied to the generative AI model, and the generated content is displayed on the interface.

[0707] Step 8:

[0708] The device performs real-time translation

[0709] Input: Text data output from the generative AI model and speech recognition system is sent to the real-time translation system.

[0710] Data calculation: A translation system (Google Translate API or DeepL API) translates the text data into the specified language.

[0711] Output: The translated text is provided to meeting participants in real time.

[0712] Specific operation: Text data is sent to the translation API and the translated text is displayed.

[0713] Step 9:

[0714] After the meeting, the server creates minutes based on the generative AI model and summarizes next action items.

[0715] Input: Recorded audio and text data is sent to the generative AI model.

[0716] Data Computation: Generative AI models analyze text data and summarize meeting minutes and next action items.

[0717] Output: The generated minutes and next action items are stored in a database.

[0718] Specific operations: Data is sent to the generative AI model, and meeting minutes and action items are automatically generated and saved.

[0719] Step 10:

[0720] The server notifies you of the minutes created and a summary of upcoming action items

[0721] Input: The generated meeting minutes and action items are retrieved from the database.

[0722] Data Processing: Notification content is formatted into a user-friendly format.

[0723] Output: The formatted notification content is sent to the user via email and / or in-app notification.

[0724] Specific operations: Retrieving data from the database, formatting the notification content, and sending the notification.

[0725] (Application example 1)

[0726] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0727] There is a demand for a system that can centrally manage meetings and briefings held within factories, including efficient progress, information sharing, and multilingual support. In addition, collaboration between robots and humans can improve productivity, and real-time speech recognition and translation, as well as the provision of appropriate questions and supplementary information, are necessary. However, conventional systems have difficulty in effectively performing these tasks, consuming a great deal of time and resources. This invention was devised to solve these problems and realize efficient and effective meeting progress.

[0728] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0729] In this invention, the server includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for saving the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting the meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; means for using the saved data to prepare for the next meeting; means for streamlining meetings and briefings held in factories and supporting collaboration between humans and robots; and means for converting the meeting content into text in real time, providing translations, and for the generation AI to provide appropriate questions and supplementary information. This enables meetings to be conducted efficiently and effectively in factories, improving productivity through collaboration between robots and humans.

[0730] "Means for inputting meeting schedules and agendas" refers to a function for registering meeting dates and topics in the system in advance.

[0731] "Means for receiving minutes of previous meetings and related materials" is a function for uploading records of past meetings and necessary materials to the system.

[0732] "Means for storing received information" refers to a function for storing uploaded minutes and materials in a database.

[0733] "Means of inputting the previous minutes and agenda into the generating AI and generating questions and supplementary information" refers to a function that inputs saved minutes and agendas into the generating AI model and automatically generates questions and supplementary information required for the meeting.

[0734] The "means for notifying the start of the conference" is a function for notifying participants of the start time of the conference.

[0735] "Means of converting meeting content into text in real time using voice recognition technology" is a function that uses technology to convert statements made during a meeting into text data in real time.

[0736] "Means for the generative AI to monitor the progress of the meeting in real time and provide appropriate questions or supplementary information" refers to a function in which the generative AI monitors the progress of the meeting and instantly provides questions or supplementary information.

[0737] "Means for providing real-time translation of meeting content using generative AI" is a function that translates meeting content in real time to ensure smooth communication between participants who speak different languages.

[0738] "A means for the generation AI to create minutes after the meeting and summarize the next action items" is a function that automatically creates minutes based on the content of the meeting and summarizes the next action items to be taken.

[0739] "Means for notifying the created minutes and a summary of the next action item" is a function for notifying participants of the created minutes and the next task.

[0740] "Means of using saved data to prepare for the next meeting" is a function that helps prepare for the next meeting by referring to past meeting records and materials.

[0741] "A means to streamline meetings and briefings held within factories and support collaboration between humans and robots" is a function that helps robots participate in meetings and information sharing held within factories so that they can proceed efficiently.

[0742] "A means of converting meeting content into text in real time, providing translation, and having a generation AI provide appropriate questions and supplementary information" refers to a function that converts statements into text data in real time, translates them as needed, and then has a generation AI provide questions and supplementary information.

[0743] The system of the present invention improves the efficiency of meetings and briefings in factories and provides a series of functions to support collaboration between humans and robots. The specific configuration and operation of the system are described below.

[0744] System Overview

[0745] 1. Preparation Phase

[0746] Users can use smartphones, tablets, or even the terminals of robots in factories to enter meeting schedules and specific agendas. For example, they can enter information such as "October 1, 2023, 10:00 AM, in Conference Room C in the factory, to hold a meeting to launch a new product line."

[0747] Users upload previous meeting minutes and related materials (e.g., production reports, safety guidelines) into the system.

[0748] This information is stored in a database by the server.

[0749] 2. Before the meeting begins

[0750] The server retrieves the meeting agenda, previous meeting minutes, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT).

[0751] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[0752] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[0753] 3. During a meeting

[0754] The user tells the system to "start a meeting."

[0755] The terminals (smartphones, factory robots) recognize the audio of the meeting in real time and convert it into text using Google Cloud Speech-to-Text.

[0756] The server monitors the progress of the meeting in real time with a generative AI. For example, when a new software product is discussed, the generative AI asks appropriate questions such as, "Please tell me more about the status of testing of the new feature."

[0757] The device uses the Google Translate API to provide real-time translation in multiple languages, ensuring that participants who speak different languages ​​can understand the content.

[0758] 4. After the meeting

[0759] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it could record that "at the meeting held on October 1, 2023, it was decided that the release date for a new product would be January 15, 2024."

[0760] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[0761] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[0762] Examples of specific examples and prompts

[0763] For example, if we take a new product release planning meeting, the flow would be as follows:

[0764] 1. A user schedules a meeting to launch a new product line in conference room C in the factory at 10:00 AM on October 1, 2023.

[0765] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "safety guidelines" to the system.

[0766] 3. The server acquires this information and inputs it into a generative AI model, which generates questions and supplementary information in advance.

[0767] 4. The conference begins and the user tells the system to "start the conference."

[0768] 5. The device converts the meeting audio into text using Google Cloud Speech-to-Text, and the server provides appropriate questions and follow-up questions using generated AI.

[0769] 6. The device uses the Google Translate API for real-time translation, enabling multilingual support.

[0770] 7. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[0771] 8. The server notifies the user of the minutes and next action items and saves them in the database.

[0772] Prompt Sentence Examples

[0773] User: "We will hold a launch meeting for the new product line in conference room C in the factory on October 1, 2023 at 10:00 AM."

[0774] Server: "Please upload the last meeting minutes, new product specifications, and safety guidelines."

[0775] User: "I uploaded it."

[0776] Server: "Do you want to start the meeting?"

[0777] User: "Yes, let's start."

[0778] Server (generative AI): "Tell me more about the challenges you're facing with your current product line."

[0779] Server (generative AI): "Please tell me the status of the commissioning of the new product line."

[0780] Server: "The meeting has ended. The minutes are being compiled..."

[0781] Server: "I've summarized our next action items: 1. Report test results for new product line. 2. Review safety guidelines."

[0782] In this way, a system is provided that enables meetings within a factory to be conducted efficiently and enables collaboration between robots and humans.

[0783] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0784] Step 1:

[0785] Users use smartphones, tablets, or robotic terminals in factories to enter meeting schedules and agendas. The entered data includes the date, time, location, and topic of the meeting. The entered information is sent to a server and stored in a database. The meeting is then prepared.

[0786] Input: Meeting schedule and agenda

[0787] Output: Meeting schedules and agendas stored in a database

[0788] Step 2:

[0789] Users upload minutes of previous meetings and related materials, such as production reports and safety guidelines, to the system, which the server then stores in a database.

[0790] Input: Minutes of previous meetings and related materials

[0791] Output: Meeting minutes and related materials stored in a database

[0792] Step 3:

[0793] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT). The generative AI automatically generates important questions and supplementary information from the meeting content and saves the output on the server.

[0794] Input: Agendas, minutes, and related materials retrieved from the database

[0795] Output: Auto-generated questions and additional information

[0796] Step 4:

[0797] When a conference starts, the user notifies the system that the conference is about to begin. The server then notifies all participants that the conference has begun.

[0798] Input: Meeting start notification

[0799] Output: Notification to participants

[0800] Step 5:

[0801] Devices (smartphones, factory robots) recognize speech during meetings in real time and convert it into text using Google Cloud Speech-to-Text. The converted speech is sent to a server, where a generating AI monitors the progress of the meeting in real time.

[0802] Input: Audio data during the meeting

[0803] Output: Real-time textualized speech data

[0804] Step 6:

[0805] The server analyzes the text of the meeting, which has been converted by the generative AI, and provides appropriate questions and supplementary information. For example, if a new product software is being discussed, the generative AI might ask, "Please tell me more about the status of testing of the new function."

[0806] Input: Translated meeting contents

[0807] Output: Questions and supplementary information generated by the generative AI

[0808] Step 7:

[0809] The device uses the Google Translate API to translate meeting content in real time and display it in a format that is easy to understand for participants who speak different languages, thereby enabling multilingual support.

[0810] Input: Translated meeting contents

[0811] Output: Real-time translated meeting content

[0812] Step 8:

[0813] After the meeting, the server uses generative AI to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0814] Input: Meeting audio and text data

[0815] Output: Generated meeting minutes

[0816] Step 9:

[0817] Based on the discussion, the generative AI summarizes action items for the next meeting, such as listing specific tasks like "complete marketing materials by the next meeting." This provides a concrete guide to action.

[0818] Input: Minutes and meeting contents

[0819] Output: List of next action items

[0820] Step 10:

[0821] The server notifies the user of the created minutes and next action items and stores them in a database, allowing the user to check the minutes and next action items, thereby streamlining preparations for the next meeting.

[0822] Input: Generated meeting minutes and action items

[0823] Output: Notify user and save to database

[0824] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0825] The system of this invention provides a series of functions for enhancing conferences through collaboration between servers, terminals, and users. Furthermore, by combining it with an emotion engine that recognizes users' emotions, more effective conference management is achieved. Below, we explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[0826] overview

[0827] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for asking questions and providing additional information, real-time translation, creating meeting minutes, summarizing upcoming action items, and recognizing user emotions with an emotion engine.

[0828] System execution example

[0829] 1. Preparation Phase

[0830] The user inputs the meeting schedule and specific agenda into the system. For example, the user inputs "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0831] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[0832] This information is stored in a database by the server.

[0833] 2. Before the meeting begins

[0834] The server retrieves the saved meeting agenda, previous minutes, and related materials from the database and inputs this information into the generative AI model.

[0835] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[0836] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[0837] 3. During a meeting

[0838] When a meeting begins, the user tells the system to "start the meeting."

[0839] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[0840] The server monitors the progress of the meeting in real time through generative AI and asks appropriate questions, such as, "Please tell me the status of software testing for the new product."

[0841] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions.

[0842] The recognized emotion data is sent to a server, and the generation AI adjusts the questions and supplementary information based on that information. For example, if the user appears confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[0843] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[0844] 4. After the meeting

[0845] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the release date for the new product will be January 15, 2024."

[0846] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[0847] Analyzes emotional data after the meeting and generates a report evaluating satisfaction and responsiveness of the meeting.

[0848] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[0849] Specific examples

[0850] For example, if we take a new product release planning meeting, the flow would be as follows:

[0851] 1. A user schedules a new product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00.

[0852] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "marketing plan" to the system.

[0853] 3. The server obtains this information and inputs it into a generative AI model to generate questions and supplementary information in advance.

[0854] 4. The conference begins and the user tells the system to "start the conference."

[0855] 5. The device converts the audio of the meeting into text, and the server provides appropriate questions and follow-up questions using generated AI.

[0856] 6. The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions, and the data is sent to the server.

[0857] 7. The server uses a generative AI to adjust questions and follow-up questions based on emotional data, and if the user appears confused, it will ask, "Could you please explain in more detail?"

[0858] 8. The device will perform real-time translation, enabling multilingual support.

[0859] 9. After the meeting, the server uses AI to create minutes and summarize the next action items.

[0860] 10. Generative AI analyzes emotional data during meetings and generates reports evaluating satisfaction and responsiveness of meetings.

[0861] 11. The server notifies the user of the minutes and next action items and saves them in the database.

[0862] In this way, the AI-Meeting Assistant system, which combines an emotion engine, provides comprehensive support from meeting preparation to progress and post-conclusion follow-up, helping to ensure fruitful meetings. By utilizing emotion data, it becomes possible to respond appropriately to the user's reactions, leading to more effective meeting management.

[0863] The processing flow will be explained below.

[0864] Step 1:

[0865] A user logs in to the system and enters the schedule and agenda for a meeting. For example, the user enters, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0866] Step 2:

[0867] A user uploads the minutes of a previous meeting and related documents to the system, such as a specification document for a new product or a marketing plan.

[0868] Step 3:

[0869] The server receives the meeting schedule, agenda, and related materials sent by the user, and then categorizes and stores the information in a database.

[0870] Step 4:

[0871] The server retrieves the saved meeting agenda, previous meeting minutes, and related materials from the database and inputs them into the generative AI model.

[0872] Step 5:

[0873] The generative AI analyzes the meeting content and generates important questions and supplementary information in advance, which is then stored on the server.

[0874] Step 6:

[0875] The server assigns roles to the generation AI, such as "questioner," "supplementary information provider," and "minutes creator."

[0876] Step 7:

[0877] When it is time for the meeting to start, the user notifies the system that the meeting should begin, for example by clicking a button.

[0878] Step 8:

[0879] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[0880] Step 9:

[0881] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions, for example, determining whether they are confused or excited.

[0882] Step 10:

[0883] The emotion engine sends the analyzed emotion data to the server, which then passes this data to the generation AI.

[0884] Step 11:

[0885] The server monitors the progress of the meeting in real time using a generative AI, which then asks appropriate questions to participants, such as, "Please tell me about the status of software testing for the new product."

[0886] Step 12:

[0887] The generative AI adjusts the user's response based on emotional data, providing further explanations or asking follow-up questions if the user is confused or does not understand.

[0888] Step 13:

[0889] Users can answer questions posed by the AI, further deepening the discussion, and the AI ​​can provide additional information as needed.

[0890] Step 14:

[0891] The device translates the meeting content in real time, allowing participants who speak different languages ​​to understand the content.

[0892] Step 15:

[0893] After the meeting, the server uses the AI ​​to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0894] Step 16:

[0895] The generative AI summarizes the next action items discussed during the meeting, listing specific tasks such as "finish marketing materials by the next meeting."

[0896] Step 17:

[0897] The server notifies the user of the created minutes and a summary of the next action items, for example, via email or in-system notifications.

[0898] Step 18:

[0899] The emotion engine analyzes post-meeting emotional data and generates a report assessing meeting satisfaction and responsiveness.

[0900] Step 19:

[0901] Use the meeting information, minutes, and summary of next action items stored in the database to prepare for the next meeting, making it more effective.

[0902] Example 2

[0903] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0904] Conventional meeting support systems required the use of multiple tools for meeting preparation, progress, and follow-up, making efficient meeting management difficult. In particular, they were unable to grasp participants' emotions and reactions in real time and provide appropriate questions or supplementary information based on that information. Communication between participants who spoke different languages ​​also posed an obstacle to meeting management.

[0905] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0906] In this invention, the server includes means for inputting meeting schedules and agendas, means for receiving minutes and related materials from the previous meeting, means for saving the received information, means for inputting the minutes and agenda from the previous meeting to a generation AI and generating questions and supplementary information, means for notifying the start of the meeting, means for converting the meeting content into text in real time using speech recognition technology, means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information, means for the generation AI to provide real-time translation of the meeting content, means for recognizing user emotions using an emotion engine and for the generation AI to adjust the questions and supplementary information based on the recognized emotion data, means for the generation AI to create minutes and summarize next action items after the meeting ends, means for analyzing emotion data and generating a report evaluating satisfaction and responsiveness, means for notifying the created minutes and summaries of next action items, and means for using the saved data to prepare for the next meeting. This provides consistent support for meetings from preparation to progress and follow-up after the end, enabling appropriate responses based on user emotions, thereby improving the efficiency and quality of meetings.

[0907] "Scheduled Meeting" means a specific schedule of a meeting to be held on a specific date and time.

[0908] "Agenda" means the list of items or topics to be discussed at a meeting.

[0909] "Meeting minutes" refers to a document that records what was discussed and what decisions were made during a meeting.

[0910] "Related Materials" means any additional information or documents related to the agenda of the meeting.

[0911] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate questions and supplemental information.

[0912] "Voice recognition technology" refers to technology that converts voice into text data.

[0913] An "emotion engine" refers to a system that recognizes emotions by analyzing a user's facial expressions and tone of voice.

[0914] "Real-time translation" refers to technology that translates conversations into different languages ​​in real time.

[0915] "Meeting proceedings" refers to the procedures and processes for smoothly running a meeting.

[0916] "Next action items" refer to specific tasks or action items to be carried out before the next meeting.

[0917] "Emotional data" refers to data that indicates the emotional state of a user.

[0918] "Evaluation Report" means a report that evaluates the satisfaction and responsiveness of a meeting based on specific criteria.

[0919] The system of the present invention provides a series of functions that consistently support the preparation, progress, and follow-up of a conference. The system operates in cooperation with a server, terminals, and users. Each function of the system is implemented as follows:

[0920] 1. Schedule meetings and enter agendas

[0921] Users input meeting schedules and agendas into the system. This information is stored in a database by the server. For example, a user might input "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[0922] 2. Uploading materials

[0923] Users upload the minutes of previous meetings and related documents (e.g., specifications for new products, marketing plans) to the system, and the server stores this information in a database.

[0924] 3. Input of meeting materials

[0925] The server retrieves the meeting agenda and related materials from the database and inputs them into the generative AI model. The generative AI model analyzes this data and generates important questions and supplemental information from the meeting content. For example, it generates a question such as, "Please tell me the status of software testing for the new product."

[0926] 4. Role assignment

[0927] The server assigns different roles to the generating AI, such as questioner, supplementary information provider, and minutes writer.

[0928] 5. Before the meeting begins

[0929] When a meeting starts, the user notifies the system that they want to start a meeting. The server then passes the prepared data to the generation AI, which prepares to support the progress of the meeting.

[0930] 6. Speech Recognition and Text Conversion

[0931] The device recognizes the audio of the meeting in real time and converts it into text format. This process records all utterances as text. For example, a utterance such as "When is the release date of the new product?" is sent to the server as text data.

[0932] 7. Real-time monitoring and questioning

[0933] The server uses generative AI to monitor the progress of the meeting in real time and provide appropriate questions and supplemental information. Based on the content of the meeting, the generative AI asks questions such as, "Please tell me about the status of software testing for our new product."

[0934] 8. Analysis by Emotion Engine

[0935] The device has a built-in emotion engine that analyzes the user's facial expressions and tone of voice to recognize their emotions. The recognized emotion data is sent to a server, and the generation AI adjusts the questions and supplementary information based on that data. For example, if the user is confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[0936] 9. Real-time translation

[0937] The device translates in real time during a meeting, ensuring that participants who speak different languages ​​can understand what is being said, for example translating what is being said in English into Japanese.

[0938] 10. Preparation of minutes

[0939] After the meeting, the server uses generative AI to create minutes based on the audio and text data from the meeting. For example, it might record, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[0940] 11. Summary of next action items

[0941] The generative AI model then summarizes the next action items based on the discussion, such as a list of specific tasks such as "finish marketing materials by the next meeting."

[0942] 12. Emotion Data Analysis and Report Generation

[0943] After the meeting, the server analyzes the emotional data and generates a report assessing satisfaction and responsiveness of the meeting.

[0944] 13. Notice and Retention

[0945] The created minutes and next action items are notified to the user by the server and stored in a database.

[0946] This allows the system to provide consistent support from meeting preparation to progress and follow-up after the meeting has ended, and enables appropriate responses based on the user's emotions, thereby improving the efficiency and quality of meetings.

[0947] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0948] Step 1:

[0949] The user enters the meeting schedule and agenda. The "meeting date and time," "location," and "topic" entered by the user are sent to the system. The server saves this information in a database. This allows the system to manage basic meeting information. Example input: "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[0950] Step 2:

[0951] Users upload the minutes of previous meetings and related materials to the system. Related materials provided by users, such as "minutes," "specifications," and "marketing plans," are sent to the system. The server receives these materials and stores them in a database. This provides background information for the meeting.

[0952] Step 3:

[0953] The server retrieves the saved meeting agenda and related materials from the database. The retrieved "agenda," "minutes," and "materials" are input into the generative AI model. The generative AI model analyzes this data and generates important questions and supplementary information from the meeting content. The generated questions are saved on the server. For example, it generates a question such as, "Please tell me the status of software testing for the new product."

[0954] Step 4:

[0955] The server assigns roles to the generating AI, such as "questioner," "supplementary information provider," and "minutes writer," which determines the role the generating AI will play during the meeting.

[0956] Step 5:

[0957] When a meeting starts, the user notifies the system that they want to start a meeting. The server receives this notification and the entire system switches to meeting mode. The server then supplies pre-prepared data to the generation AI and begins supporting the progress of the meeting.

[0958] Step 6:

[0959] The device recognizes the audio of the meeting in real time and converts it into text. The input audio data is converted into "text data" using the device's voice recognition technology. The converted text data is sent to the server and stored. For example, the audio "When is the release date for the new product?" is converted into text.

[0960] Step 7:

[0961] The server uses generative AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information. It analyzes what is being said in the meeting in real time and asks questions based on the content of the meeting. For example, it might ask, "Please tell me about the status of software testing for our new product."

[0962] Step 8:

[0963] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions. The analyzed "emotion data" is sent from the device to a server. The server passes the received emotion data to the generation AI, which then adjusts the questions and supplementary information based on the emotion data. For example, if the user seems confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[0964] Step 9:

[0965] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand the content. Based on the input "voice data" and "text data," "translated text" is generated and provided to other participants. For example, content spoken in English can be translated into Japanese.

[0966] Step 10:

[0967] After the meeting ends, the server uses a generation AI to create minutes based on the audio and text data from the meeting. The generation AI analyzes the input audio and text data and generates minutes text. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the release date for the new product will be January 15, 2024."

[0968] Step 11:

[0969] The generative AI model summarizes next action items based on the content of the discussion. It analyzes the "minutes text" as input and generates "next action items." For example, it lists specific tasks such as "Complete marketing materials by the next meeting."

[0970] Step 12:

[0971] After the meeting ends, the server analyzes the emotional data and generates a report evaluating the satisfaction and responsiveness of the meeting. Based on the input "emotional data," a "satisfaction evaluation report" and a "responsiveness evaluation report" are generated. This allows the quality of the meeting to be objectively evaluated.

[0972] Step 13:

[0973] The server notifies the user of the minutes and next action items, and stores them in a database, which can be used to prepare for the next meeting.

[0974] (Application example 2)

[0975] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0976] Current meeting management systems often require manual preparation, progress, and follow-up after a meeting, which is labor-intensive and time-consuming. They also face challenges in being unable to meet advanced requirements, such as conducting meetings in multiple languages ​​or recognizing participants' emotions. In particular, there is a lack of systems that can properly manage workers' emotions and stress in production line work environments. Therefore, there is a need for a system that can simultaneously improve meeting efficiency and consider the emotions of workers.

[0977] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0978] In this invention, the server includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for saving the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting the meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; an emotion engine that recognizes participants' emotions during the meeting and analyzes the emotion data to adjust questions and supplementary information; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; and means for using the saved data to prepare for the next meeting. This provides comprehensive support for meetings from preparation to progress and follow-up after the meeting, and by utilizing emotion data, it is possible to appropriately manage worker reactions and stress, enabling effective meeting management and improving the working environment.

[0979] The "means for inputting meeting schedules and agendas" is a function that allows a user to input schedule information such as the date and time of a meeting and the agenda into the system.

[0980] "Means for receiving minutes of the previous meeting and related materials" is a function for uploading minutes created at the previous meeting and materials related to the meeting to the system.

[0981] "Means for storing received information" refers to a function for storing meeting information and materials entered or uploaded into the system in a database.

[0982] "Means of inputting the previous meeting minutes and agenda into the generating AI and generating questions and supplementary information" refers to a function that inputs the previous meeting minutes and agenda into the AI ​​and automatically generates important questions and supplementary information based on them.

[0983] The "means for notifying the start of a conference" is a function for notifying the system when a user starts a conference.

[0984] "Means for converting meeting content into text in real time using voice recognition technology" is a function that uses voice recognition technology to convert voices spoken during a meeting into text in real time.

[0985] "Means for generative AI to monitor the progress of the meeting in real time and provide appropriate questions or supplementary information" refers to a function that monitors the progress of the meeting in real time and allows the AI ​​to provide questions or supplementary information at the appropriate time.

[0986] "Means for providing real-time translation of meeting content using generative AI" refers to a function that uses AI to translate meeting content in real time so that meeting participants who speak different languages ​​can understand the content.

[0987] "Means for adjusting questions and supplementary information by analyzing emotional data, including an emotion engine that recognizes participants' emotions during the meeting" refers to a function that uses an emotion engine that analyzes participants' facial expressions and tone of voice to obtain emotional data during the meeting, and then uses the AI ​​to adjust questions and supplementary information based on that data.

[0988] "A means for the generation AI to create minutes after the meeting and summarize action items for the next meeting" is a function in which the AI ​​automatically creates minutes based on the meeting content after the meeting and summarizes action items for the next meeting.

[0989] The "means for notifying the user of the created minutes and a summary of the next action item" is a function for notifying the user of the created minutes and a summary of the next action item.

[0990] "Means for using saved data in preparation for the next meeting" is a function for utilizing past meeting data saved in the database in preparation for the next meeting.

[0991] The system for implementing this invention is configured to provide various functions through cooperation between a server, a terminal, and a user. This system includes programs for realizing the following functions based on the claims.

[0992] First, the user enters the meeting schedule and agenda, and uploads the minutes of the previous meeting and related materials to the system. This information is sent to the server via the device and stored in the server's database. The server then inputs the stored information into the generation AI, which automatically generates questions and supplementary information based on the meeting content.

[0993] At the start of a meeting, the user uses their device to announce "start meeting." The device then converts the meeting content into text in real time using voice recognition technology, and the generation AI provides real-time questions and supplementary information. Furthermore, during the meeting, an emotion engine recognizes the emotions of participants and sends this data to the server. The generation AI then adjusts the questions and supplementary information based on the emotion data, optimizing the progress of the meeting. The generation AI also provides real-time translation for multilingual meeting participants.

[0994] After the meeting ends, the server uses generative AI to create minutes based on the audio and text data of the meeting and summarizes the next action items. This information is notified to the user and saved in a database. The saved data can be used to prepare for the next meeting.

[0995] As a specific example, let's imagine a new product release planning meeting, and the flow would be as follows:

[0996] 1. A user schedules a new product launch meeting to be held in the conference room on October 1, 2023 at 10:00 AM.

[0997] 2. Upload previous meeting minutes, new product specifications, and marketing plans into the system.

[0998] 3. The server obtains this information and inputs it into a generative AI model to generate questions and supplementary information in advance.

[0999] 4. When the conference starts, the user tells the system to "start the conference."

[1000] 5. The device converts the audio of the meeting into text, and the generative AI provides appropriate questions and follow-up comments.

[1001] 6. The device's built-in emotion engine analyzes the user's facial expressions and tone of voice and sends the emotion data to the server.

[1002] 7. Based on the emotion data, the server will ask questions and provide further explanations. If the user is confused, the AI ​​will ask questions such as, "Could you explain more specifically?"

[1003] 8. The device will translate the meeting content in real time, enabling multilingual support.

[1004] 9. After the meeting, the server uses AI to create minutes and summarize the next action items.

[1005] 10. The generated minutes and next action items are notified to the user and saved in the database.

[1006] The hardware used includes smartphones and head-mounted displays, and the software used includes the speech_recognition library (speech recognition) and the transformers library (generative AI and emotion recognition).

[1007] Example prompt sentence:

[1008] "A system that recognizes speech and analyzes emotions in real time during new product launch meetings"

[1009] "I want to automatically generate questions based on the presentation content and save them as meeting minutes."

[1010] In this way, the system improves meeting efficiency and the working environment.

[1011] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1012] Step 1:

[1013] The user inputs the meeting schedule and agenda. At this stage, the user inputs information such as "The new product launch meeting will be held on October 1, 2023 at 10:00" through the system interface. This obtains the meeting date and agenda as input data.

[1014] Step 2:

[1015] The user uploads the minutes of the previous meeting and related materials to the system. In this step, the user sends the minutes of the previous meeting, the specifications of the new product, the marketing plan, and other related materials to the system, which then receives these materials as input data.

[1016] Step 3:

[1017] The server stores the received information. The server accumulates the meeting schedule, agenda, and related materials obtained in steps 1 and 2 in a database. This stores the received information in the database and saves the accumulated data.

[1018] Step 4:

[1019] The server inputs the previous meeting minutes and agenda into the generation AI, which then generates questions and supplementary information. The server inputs the previous meeting minutes and agenda saved from the database into the generation AI model. The generation AI analyzes this data and automatically generates important questions and supplementary information related to the meeting content. The generated questions and supplementary information are obtained as output data.

[1020] Step 5:

[1021] The user notifies the system of the start of the conference. The user inputs "start conference" into the system via the terminal, and the conference starts. This acquires the conference start notification as input data.

[1022] Step 6:

[1023] The device uses speech recognition technology to convert the contents of the meeting into text in real time. The device picks up the meeting audio with its built-in microphone and converts the audio into text using the speech_recognition library. At this stage, the real-time converted text becomes the output data.

[1024] Step 7:

[1025] The server uses generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information. The server monitors the text data of the meeting content generated in step 6 and adds automatically generated questions and supplementary information using generation AI at the appropriate time. As a result, the questions and supplementary information provided become the output.

[1026] Step 8:

[1027] The device uses AI generation to provide real-time translation of the meeting content. The device translates text data in real time to support multiple languages, allowing participants who speak multiple languages ​​to understand the content. At this stage, the real-time translated meeting content is obtained as the output.

[1028] Step 9:

[1029] The device uses an emotion engine to recognize participants' emotions during the meeting and analyzes the emotion data to adjust questions and follow-up information. The device's built-in camera and microphone analyze facial expressions and vocal tone to recognize emotions. Based on this analysis, the emotion data is sent to a server, which then uses generative AI to adjust questions and follow-up information. This allows questions and follow-up information to be output that are adapted to the emotions.

[1030] Step 10:

[1031] The server uses the generation AI to create meeting minutes and summarize action items for the next meeting. After the meeting ends, the server uses the meeting content text data from step 6 and the emotion data from step 9 to automatically generate meeting minutes using the generation AI. In addition, based on the matters discussed during the meeting, the server summarizes action items for the next meeting. This outputs specific action items for the next meeting.

[1032] Step 11:

[1033] The server notifies the user of the created minutes and next action items. The server notifies the user of the automatically generated minutes and next action items and also stores them in a database. The final minutes and action items are notified to the user and are also stored as preparation data for the next meeting.

[1034] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1035] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1036] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1037] [Third embodiment]

[1038] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1039] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1040] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1041] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1042] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1043] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1044] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1045] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1046] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1047] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1048] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1049] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1050] The system of this invention provides a series of functions for enhancing conferences through collaboration between the server, terminals, and users. Below, we will explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[1051] overview

[1052] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for questions and supplementary information, real-time translation, meeting minutes, and a summary of upcoming action items.

[1053] System execution example

[1054] 1. Preparation Phase

[1055] The user inputs the meeting schedule and specific agenda into the system. For example, the user inputs "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1056] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[1057] This information is stored in a database by the server.

[1058] 2. Before the meeting begins

[1059] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into the generative AI model.

[1060] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[1061] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[1062] 3. During a meeting

[1063] When a meeting begins, the user tells the system to "start the meeting."

[1064] The device recognizes the audio of the meeting in real time and converts it into text.

[1065] The server monitors the progress of the meeting in real time with a generative AI. For example, if a new product software is being discussed, the generative AI will ask appropriate questions, such as, "Could you please tell me more about the status of testing of the new feature?"

[1066] The user's statements are recorded, and the generating AI provides supplementary information as needed.

[1067] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[1068] 4. After the meeting

[1069] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1070] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[1071] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[1072] Specific examples

[1073] For example, if we take a new product release planning meeting, the flow would be as follows:

[1074] 1. A user schedules a new product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00.

[1075] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "marketing plan" to the system.

[1076] 3. The server obtains this information and inputs it into the generative AI model, which generates questions and supplementary information in advance.

[1077] 4. The conference begins and the user notifies the system to "start the conference."

[1078] 5. The device converts the audio of the meeting into text, and the server uses generated AI to provide appropriate questions and additional information.

[1079] 6. The device performs real-time translation, enabling multilingual support.

[1080] 7. After the meeting, the server uses AI to create minutes and summarize next action items.

[1081] 8. The server notifies the user of the minutes and next action items and stores them in the database.

[1082] In this way, the AI-Meeting Assistant system provides comprehensive support from meeting preparation to progress and follow-up after the meeting has ended, helping to ensure fruitful meetings.

[1083] The processing flow will be explained below.

[1084] Step 1:

[1085] A user logs in to the system and enters the schedule and agenda for a meeting. For example, the user enters, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1086] Step 2:

[1087] A user uploads the minutes of a previous meeting and related documents to the system, such as a specification document for a new product or a marketing plan.

[1088] Step 3:

[1089] The server receives the meeting schedule, agenda, and related materials sent by the user, and then categorizes and stores the information in a database.

[1090] Step 4:

[1091] The server retrieves the saved meeting agenda, previous meeting minutes, and related materials from the database and inputs them into the generative AI model.

[1092] Step 5:

[1093] The generative AI analyzes the meeting content and generates important questions and supplementary information in advance, which is then stored on the server.

[1094] Step 6:

[1095] The server assigns roles to the generation AI, such as "questioner," "supplementary information provider," and "minutes creator."

[1096] Step 7:

[1097] When it is time for the meeting to start, the user notifies the system that the meeting should begin, for example by clicking a button.

[1098] Step 8:

[1099] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[1100] Step 9:

[1101] The server monitors the progress of the meeting in real time through a generative AI, which then asks appropriate questions to participants, such as, "Please tell me about the status of testing the software for our new product."

[1102] Step 10:

[1103] Users can answer questions posed by the AI, further deepening the discussion, and the AI ​​can provide additional information as needed.

[1104] Step 11:

[1105] The device translates the meeting content in real time, allowing participants who speak different languages ​​to understand the content.

[1106] Step 12:

[1107] After the meeting, the server uses the AI ​​to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1108] Step 13:

[1109] The generative AI summarizes the next action items discussed during the meeting, listing specific tasks such as "finish marketing materials by the next meeting."

[1110] Step 14:

[1111] The server notifies the user of the created minutes and a summary of the next action items, for example, via email or in-system notifications.

[1112] Step 15:

[1113] Use the meeting information, minutes, and summary of next action items stored in the database to prepare for the next meeting, making it more effective.

[1114] Example 1

[1115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1116] In conventional conferencing systems, meeting preparation, proceeding, and minutes creation are done manually, which requires time and effort. Furthermore, difficulties with real-time speech recognition and translation, and the provision of appropriate questions and supplementary information, result in a decline in meeting efficiency and quality. This invention aims to solve these problems by automating meeting preparation, proceeding, minutes creation, and summarizing next action items.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1118] In this invention, the server includes means for inputting meeting schedules and agendas, means for receiving minutes and related materials from the previous meeting, means for storing the received information, means for inputting the minutes and agenda from the previous meeting into a generative AI model and generating questions and supplementary information, means for notifying the start of the meeting, means for converting the meeting content into text in real time using speech recognition technology, means for the generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information, means for the generative AI model to provide real-time translation of the meeting content, means for the generative AI model to create minutes and summarize next action items after the meeting ends, means for notifying the created minutes and summaries of next action items, and means for using the stored data to prepare for the next meeting. This makes it possible to streamline a series of meeting processes, such as preparing for the meeting, conducting the meeting, creating minutes, and summarizing next action items, thereby improving the quality and productivity of meetings.

[1119] A "meeting schedule and agenda entry means" is an interface through which a user enters meeting dates, times, locations, and agendas into the system.

[1120] The "means for receiving minutes of previous meetings and related materials" is an interface that allows users to upload records of previous meetings and documents related to meetings to the system.

[1121] The "means for storing received information" is a function for storing the meeting schedule and agenda entered by the user and the uploaded materials in a database.

[1122] "Means of inputting previous meeting minutes and agendas into a generative AI model to generate questions and follow-up information" refers to the process and techniques for inputting stored meeting data into a generative AI model to generate important questions and follow-up information.

[1123] The "means for notifying the start of a conference" is an operation interface that allows a user to notify the system of the start of a conference.

[1124] "Means for converting meeting content into text in real time using speech recognition technology" refers to technology and devices for converting speech made during a meeting from voice to text in real time.

[1125] "Means for a generative AI model to monitor the progress of a meeting in real time and provide appropriate questions and supplementary information" refers to a technology that enables a generative AI model to analyze the content of a meeting, ask appropriate questions on the spot, and provide supplementary information.

[1126] "Means for providing real-time translation of meeting content using generative AI models" refers to technologies and devices that translate speech in real time during a meeting so that all participants who speak different languages ​​can understand the content.

[1127] "Means for a generative AI model to create minutes after a meeting and summarize action items for the next meeting" refers to a technology that analyzes the contents of a meeting from recorded audio and text data, and generates minutes and action items to be addressed at the next meeting.

[1128] The "means for notifying the user of the created minutes and a summary of the next action item" is a function for notifying the user of the created minutes and action items.

[1129] The "means for utilizing stored data in preparation for the next meeting" refers to the process and technology for utilizing stored meeting data in preparation for the next meeting.

[1130] The system of the present invention provides a series of functions for efficiently proceeding and managing a conference through collaboration between a server, terminals, and users. An embodiment of the program of this system will be described in detail below.

[1131] System Configuration

[1132] server

[1133] The server plays a central role in the system and provides the following functions:

[1134] 1. Database management: Store, retrieve, and manage meeting schedules, agendas, minutes, related materials, etc. MySQL or PostgreSQL is used as the RDBMS.

[1135] 2. Generative AI model interface: Input data into a generative AI model (e.g., OpenAI GPT-4) and have it generate questions and supplemental information. The generative AI model operates by receiving the following prompts:

[1136] "Generate important questions and supporting information to discuss at the next meeting."

[1137] 3. Real-time data processing: Audio and text data from meetings are processed in real time and passed to the generative AI model.

[1138] Terminal

[1139] The device is used at the meeting site and provides the following features:

[1140] 1. Speech Recognition: Converts meeting audio into text in real time using Google Cloud Speech-to-Text and Azure Speech Services.

[1141] 2. Real-time translation: Translates meeting content in real time, enabling multilingual support. Translation is performed using the Google Translate API and DeepL API.

[1142] 3. User operation interface: Provides a UI for users to start meetings and upload materials, using web forms or dedicated apps.

[1143] User

[1144] Users operate the system and fulfill the following roles:

[1145] 1. Entering meeting schedules and agendas: Enter the meeting date, time, and agenda into the system. For example, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[1146] 2. Upload related materials: Upload the minutes of the previous meeting and related materials using drag-and-drop or file selection dialogs.

[1147] 3. Conference start notification: Notify the system that a conference is about to begin.

[1148] Specific examples

[1149] Take the example of a new product release planning meeting:

[1150] 1. Preparation Phase

[1151] The user enters "New product launch meeting to be held in conference room C at 10:00 on October 1, 2023" into the web form and sets up the meeting schedule in the system.

[1152] Users upload documents such as "minutes of the previous meeting," "specifications for a new product," and "marketing plans" into the system.

[1153] The server stores this information in a database.

[1154] 2. Before the meeting begins

[1155] The server retrieves the meeting agenda and related materials from the database and inputs them into the generative AI model.

[1156] The generative AI model generates key questions and supplementary information based on the prompt sentence and stores it on the server.

[1157] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generative AI model.

[1158] 3. During a meeting

[1159] When a meeting begins, the user tells the system to "start the meeting."

[1160] The device converts the meeting audio into text in real time and sends it to a generative AI model.

[1161] The server uses a generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information.

[1162] The device provides real-time translation, allowing all participants, even those speaking different languages, to understand the content of the meeting.

[1163] 4. After the meeting

[1164] The server creates meeting minutes based on the voice and text data recorded by the generative AI model. For example, it might record, "At the meeting held on October 1, 2023, the release date for the new product was decided to be January 15, 2024."

[1165] The generative AI model summarizes action items for the next meeting based on the discussion, listing specific tasks such as "finish marketing materials by the next meeting."

[1166] The server notifies the user of the created minutes and next action items and stores them in a database.

[1167] In this way, the form for implementing the invention improves the quality and productivity of meetings by coordinating the server, terminals, and users to consistently improve the efficiency of meeting preparation, progress, and follow-up after the meeting has ended.

[1168] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1169] Step 1:

[1170] Users enter meeting schedules and agendas

[1171] Input: The user enters the date, time, location, and agenda of the meeting into a web form or dedicated app. For example, they might enter, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[1172] Data processing: The input information is sent to the server as text data. The meeting ID, date, time, location, and agenda are parsed and formatted as structured data.

[1173] Output: The formatted data is stored in an internal database.

[1174] Specific behavior: A front-end component (e.g., HTML form, JavaScript) is executed to receive form input and a POST request is executed to send the input to the server.

[1175] Step 2:

[1176] Users upload previous meeting minutes and related materials

[1177] Input: Users upload files such as minutes from a previous meeting, specifications for a new product, or a marketing plan to the system using drag-and-drop operations or file selection dialogs.

[1178] Data processing: Uploaded files are received by the server and stored in a location appropriate for the file format. Metadata (file name, upload date, etc.) is also generated.

[1179] Output: The file body and metadata are saved to the database and file storage.

[1180] Specific behavior: A file selection dialog on the front end, a file upload API endpoint, and a storage system for saving files (e.g. AWS S3, Google Cloud Storage) are used.

[1181] Step 3:

[1182] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database.

[1183] Input: Before a meeting starts, the system queries the database based on the meeting ID entered by the user.

[1184] Data processing: The necessary data (agenda, minutes, related materials) is retrieved from the database and formatted to be input into the generative AI model.

[1185] Output: The formatted data is temporarily stored in memory.

[1186] Specific operations: Executes SQL queries, formats and transforms data, and caches retrieved data in memory.

[1187] Step 4:

[1188] Input the previous meeting minutes and agenda into the generative AI model to generate questions and supplemental information

[1189] Input: Formatted minutes and agenda are sent to the generative AI model with the prompt, "Generate key questions and supporting information to discuss at the next meeting."

[1190] Data Computation: Generative AI models analyze text data and generate questions and follow-up information based on specified prompts.

[1191] Output: The generated questions and supplementary information are returned in text format to the server and stored in an appropriate database.

[1192] Specific operation: An API request is sent to the generative AI model, and the generated content is saved.

[1193] Step 5:

[1194] User tells the system to "start a meeting"

[1195] Input: To start a meeting, the user clicks the "Start Meeting" button on the dedicated app or web interface.

[1196] Data processing: The server notifies the entire system that the conference has started and performs the necessary initialization processing.

[1197] Output: The conference state is updated to "Started" and relevant system components are notified.

[1198] Specific operation: The button click event is processed, the conference state management system is updated, and each component is notified.

[1199] Step 6:

[1200] The device recognizes the audio of the meeting in real time and converts it into text.

[1201] Input: Voice data collected by the device's microphone during the meeting is sent to the voice recognition system.

[1202] Data processing: A speech recognition system (such as Google Cloud Speech-to-Text or Azure Speech Services) analyzes the voice data and converts it into text data in real time.

[1203] Output: The text of the remarks is sent to the server and saved as a meeting record.

[1204] Specific operations: Collects voice data, calls voice recognition API, and sends and saves text data.

[1205] Step 7:

[1206] Generative AI models monitor meeting progress in real time and provide relevant questions and follow-up information

[1207] Input: Real-time transcribed meeting content is fed into the generative AI model.

[1208] Data Computation: The generative AI model analyzes the supplied text data and generates appropriate questions and supplemental information.

[1209] Output: Generated questions and supplemental information are provided to meeting participants in real time.

[1210] Specific operation: Real-time data is supplied to the generative AI model, and the generated content is displayed on the interface.

[1211] Step 8:

[1212] The device performs real-time translation

[1213] Input: Text data output from the generative AI model and speech recognition system is sent to the real-time translation system.

[1214] Data calculation: A translation system (Google Translate API or DeepL API) translates the text data into the specified language.

[1215] Output: The translated text is provided to meeting participants in real time.

[1216] Specific operation: Text data is sent to the translation API and the translated text is displayed.

[1217] Step 9:

[1218] After the meeting, the server creates minutes based on the generative AI model and summarizes next action items.

[1219] Input: Recorded audio and text data is sent to the generative AI model.

[1220] Data Computation: Generative AI models analyze text data and summarize meeting minutes and next action items.

[1221] Output: The generated minutes and next action items are stored in a database.

[1222] Specific operations: Data is sent to the generative AI model, and meeting minutes and action items are automatically generated and saved.

[1223] Step 10:

[1224] The server notifies you of the minutes created and a summary of upcoming action items

[1225] Input: The generated meeting minutes and action items are retrieved from the database.

[1226] Data Processing: Notification content is formatted into a user-friendly format.

[1227] Output: The formatted notification content is sent to the user via email and / or in-app notification.

[1228] Specific operations: Retrieving data from the database, formatting the notification content, and sending the notification.

[1229] (Application example 1)

[1230] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1231] There is a demand for a system that can centrally manage meetings and briefings held within factories, including efficient progress, information sharing, and multilingual support. In addition, collaboration between robots and humans can improve productivity, and real-time speech recognition and translation, as well as the provision of appropriate questions and supplementary information, are necessary. However, conventional systems have difficulty in effectively performing these tasks, consuming a great deal of time and resources. This invention was devised to solve these problems and realize efficient and effective meeting progress.

[1232] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1233] In this invention, the server includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for saving the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting the meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; means for using the saved data to prepare for the next meeting; means for streamlining meetings and briefings held in factories and supporting collaboration between humans and robots; and means for converting the meeting content into text in real time, providing translations, and for the generation AI to provide appropriate questions and supplementary information. This enables meetings to be conducted efficiently and effectively in factories, improving productivity through collaboration between robots and humans.

[1234] "Means for inputting meeting schedules and agendas" refers to a function for registering meeting dates and topics in the system in advance.

[1235] "Means for receiving minutes of previous meetings and related materials" is a function for uploading records of past meetings and necessary materials to the system.

[1236] "Means for storing received information" refers to a function for storing uploaded minutes and materials in a database.

[1237] "Means of inputting the previous minutes and agenda into the generating AI and generating questions and supplementary information" refers to a function that inputs saved minutes and agendas into the generating AI model and automatically generates questions and supplementary information required for the meeting.

[1238] The "means for notifying the start of the conference" is a function for notifying participants of the start time of the conference.

[1239] "Means of converting meeting content into text in real time using voice recognition technology" is a function that uses technology to convert statements made during a meeting into text data in real time.

[1240] "Means for the generative AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information" refers to a function in which the generative AI monitors the progress of the meeting and instantly provides questions and supplementary information.

[1241] "Means for providing real-time translation of meeting content using generative AI" is a function that translates meeting content in real time to ensure smooth communication between participants who speak different languages.

[1242] "A means for the generation AI to create minutes after the meeting and summarize the next action items" is a function that automatically creates minutes based on the content of the meeting and summarizes the next action items to be taken.

[1243] "Means for notifying the created minutes and a summary of the next action item" is a function for notifying participants of the created minutes and the next task.

[1244] "Means of using saved data to prepare for the next meeting" is a function that helps prepare for the next meeting by referring to past meeting records and materials.

[1245] "A means to streamline meetings and briefings held within factories and support collaboration between humans and robots" is a function that helps robots participate in meetings and information sharing held within factories so that they can proceed efficiently.

[1246] "A means of converting meeting content into text in real time, providing translation, and having a generation AI provide appropriate questions and supplementary information" refers to a function that converts statements into text data in real time, translates them as needed, and then has a generation AI provide questions and supplementary information.

[1247] The system of the present invention improves the efficiency of meetings and briefings in factories and provides a series of functions to support collaboration between humans and robots. The specific configuration and operation of the system are described below.

[1248] System Overview

[1249] 1. Preparation Phase

[1250] Users can use smartphones, tablets, or even the terminals of robots in the factory to enter meeting schedules and specific agendas. For example, they can enter information such as "October 1, 2023, 10:00 AM, in Conference Room C in the factory, to hold a meeting to launch a new product line."

[1251] Users upload previous meeting minutes and related materials (e.g., production reports, safety guidelines) into the system.

[1252] This information is stored in a database by the server.

[1253] 2. Before the meeting begins

[1254] The server retrieves the meeting agenda, previous meeting minutes, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT).

[1255] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[1256] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[1257] 3. During a meeting

[1258] The user tells the system to "start a meeting."

[1259] The terminals (smartphones, factory robots) recognize the audio of the meeting in real time and convert it into text using Google Cloud Speech-to-Text.

[1260] The server monitors the progress of the meeting in real time with a generative AI. For example, when a new software product is discussed, the generative AI asks appropriate questions such as, "Could you please tell me more about the status of testing of the new feature?"

[1261] The device uses the Google Translate API to provide real-time translation in multiple languages, ensuring that participants who speak different languages ​​can understand the content.

[1262] 4. After the meeting

[1263] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it could record that "at the meeting held on October 1, 2023, it was decided that the release date for a new product would be January 15, 2024."

[1264] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[1265] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[1266] Examples of specific examples and prompts

[1267] For example, if we take a new product release planning meeting, the flow would be as follows:

[1268] 1. A user schedules a meeting to launch a new product line in conference room C in the factory at 10:00 AM on October 1, 2023.

[1269] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "safety guidelines" to the system.

[1270] 3. The server acquires this information and inputs it into a generative AI model, which generates questions and supplementary information in advance.

[1271] 4. The conference begins and the user tells the system to "start the conference."

[1272] 5. The device converts the meeting audio into text using Google Cloud Speech-to-Text, and the server provides appropriate questions and follow-up questions using generated AI.

[1273] 6. The device uses the Google Translate API for real-time translation, enabling multilingual support.

[1274] 7. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[1275] 8. The server notifies the user of the minutes and next action items and saves them in the database.

[1276] Prompt Sentence Examples

[1277] User: "We will hold a launch meeting for the new product line in conference room C in the factory on October 1, 2023 at 10:00 AM."

[1278] Server: "Please upload the last meeting minutes, new product specifications, and safety guidelines."

[1279] User: "I uploaded it."

[1280] Server: "Do you want to start the meeting?"

[1281] User: "Yes, let's start."

[1282] Server (generative AI): "Tell me more about the challenges you're facing with your current product line."

[1283] Server (generative AI): "Please tell me the status of the commissioning of the new product line."

[1284] Server: "The meeting has ended. The minutes are being compiled..."

[1285] Server: "I've summarized our next action items: 1. Report test results for new product line. 2. Review safety guidelines."

[1286] In this way, a system is provided that enables meetings within a factory to be conducted efficiently and enables collaboration between robots and humans.

[1287] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1288] Step 1:

[1289] Users use smartphones, tablets, or robotic terminals in factories to enter meeting schedules and agendas. The entered data includes the date, time, location, and topic of the meeting. The entered information is sent to a server and stored in a database. The meeting is then prepared.

[1290] Input: Meeting schedule and agenda

[1291] Output: Meeting schedules and agendas stored in a database

[1292] Step 2:

[1293] Users upload minutes of previous meetings and related materials, such as production reports and safety guidelines, to the system, which the server then stores in a database.

[1294] Input: Minutes of previous meetings and related materials

[1295] Output: Meeting minutes and related materials stored in a database

[1296] Step 3:

[1297] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT). The generative AI automatically generates important questions and supplementary information from the meeting content and saves the output on the server.

[1298] Input: Agendas, minutes, and related materials retrieved from the database

[1299] Output: Auto-generated questions and additional information

[1300] Step 4:

[1301] When a conference starts, the user notifies the system that the conference is about to begin. The server then notifies all participants that the conference has begun.

[1302] Input: Meeting start notification

[1303] Output: Notification to participants

[1304] Step 5:

[1305] Devices (smartphones, factory robots) recognize speech during meetings in real time and convert it into text using Google Cloud Speech-to-Text. The converted speech is sent to a server, where a generating AI monitors the progress of the meeting in real time.

[1306] Input: Audio data during the meeting

[1307] Output: Real-time textualized speech data

[1308] Step 6:

[1309] The server analyzes the text of the meeting, which has been converted by the generative AI, and provides appropriate questions and supplementary information. For example, if a new product software is being discussed, the generative AI might ask, "Please tell me more about the status of testing of the new function."

[1310] Input: Translated meeting contents

[1311] Output: Questions and supplementary information generated by the generative AI

[1312] Step 7:

[1313] The device uses the Google Translate API to translate meeting content in real time and display it in a format that is easy to understand for participants who speak different languages, thereby enabling multilingual support.

[1314] Input: Translated meeting contents

[1315] Output: Real-time translated meeting content

[1316] Step 8:

[1317] After the meeting, the server uses generative AI to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1318] Input: Meeting audio and text data

[1319] Output: Generated meeting minutes

[1320] Step 9:

[1321] Based on the discussion, the generative AI summarizes action items for the next meeting, such as listing specific tasks like "complete marketing materials by the next meeting." This provides a concrete guide to action.

[1322] Input: Minutes and meeting contents

[1323] Output: List of next action items

[1324] Step 10:

[1325] The server notifies the user of the created minutes and next action items and stores them in a database, allowing the user to check the minutes and next action items, thereby streamlining preparations for the next meeting.

[1326] Input: Generated meeting minutes and action items

[1327] Output: Notify user and save to database

[1328] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1329] The system of this invention provides a series of functions for enhancing conferences through collaboration between servers, terminals, and users. Furthermore, by combining it with an emotion engine that recognizes users' emotions, more effective conference management is achieved. Below, we explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[1330] overview

[1331] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for asking questions and providing additional information, real-time translation, creating meeting minutes, summarizing upcoming action items, and recognizing user emotions with an emotion engine.

[1332] System execution example

[1333] 1. Preparation Phase

[1334] The user inputs the meeting schedule and specific agenda into the system. For example, the user might input "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1335] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[1336] This information is stored in a database by the server.

[1337] 2. Before the meeting begins

[1338] The server retrieves the saved meeting agenda, previous minutes, and related materials from the database and inputs this information into the generative AI model.

[1339] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[1340] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[1341] 3. During a meeting

[1342] When a meeting begins, the user tells the system to "start the meeting."

[1343] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[1344] The server monitors the progress of the meeting in real time through generative AI and asks appropriate questions, such as, "Please tell me the status of software testing for the new product."

[1345] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions.

[1346] The recognized emotion data is sent to a server, and the generation AI adjusts the questions and supplementary information based on that information. For example, if the user appears confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[1347] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[1348] 4. After the meeting

[1349] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the release date for the new product will be January 15, 2024."

[1350] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[1351] Analyzes emotional data after the meeting and generates a report evaluating satisfaction and responsiveness of the meeting.

[1352] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[1353] Specific examples

[1354] For example, if we take a new product release planning meeting, the flow would be as follows:

[1355] 1. A user schedules a new product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00.

[1356] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "marketing plan" to the system.

[1357] 3. The server obtains this information and inputs it into a generative AI model to generate questions and supplementary information in advance.

[1358] 4. The conference begins and the user tells the system to "start the conference."

[1359] 5. The device converts the audio of the meeting into text, and the server provides appropriate questions and follow-up questions using generated AI.

[1360] 6. The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize emotions, and the data is sent to the server.

[1361] 7. The server uses a generative AI to adjust questions and follow-up questions based on emotional data, and if the user appears confused, it will ask, "Could you please explain in more detail?"

[1362] 8. The device will perform real-time translation, enabling multilingual support.

[1363] 9. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[1364] 10. Generative AI analyzes emotional data during meetings and generates reports evaluating satisfaction and responsiveness of meetings.

[1365] 11. The server notifies the user of the minutes and next action items and saves them in the database.

[1366] In this way, the AI-Meeting Assistant system, which combines an emotion engine, provides comprehensive support from meeting preparation to progress and post-conclusion follow-up, helping to ensure fruitful meetings. By utilizing emotion data, it becomes possible to respond appropriately to the user's reactions, leading to more effective meeting management.

[1367] The processing flow will be explained below.

[1368] Step 1:

[1369] A user logs in to the system and enters the schedule and agenda for a meeting. For example, the user enters, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1370] Step 2:

[1371] A user uploads the minutes of a previous meeting and related documents to the system, such as a specification document for a new product or a marketing plan.

[1372] Step 3:

[1373] The server receives the meeting schedule, agenda, and related materials sent by the user, and then categorizes and stores the information in a database.

[1374] Step 4:

[1375] The server retrieves the saved meeting agenda, previous meeting minutes, and related materials from the database and inputs them into the generative AI model.

[1376] Step 5:

[1377] The generative AI analyzes the meeting content and generates important questions and supplementary information in advance, which is then stored on the server.

[1378] Step 6:

[1379] The server assigns roles to the generation AI, such as "questioner," "supplementary information provider," and "minutes creator."

[1380] Step 7:

[1381] When it is time for the meeting to start, the user notifies the system that the meeting should begin, for example by clicking a button.

[1382] Step 8:

[1383] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[1384] Step 9:

[1385] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions, for example, determining whether they are confused or excited.

[1386] Step 10:

[1387] The emotion engine sends the analyzed emotion data to the server, which then passes this data to the generation AI.

[1388] Step 11:

[1389] The server monitors the progress of the meeting in real time using a generative AI, which then asks appropriate questions to participants, such as, "Please tell me about the status of testing the software for our new product."

[1390] Step 12:

[1391] The generative AI adjusts the user's response based on emotional data, providing further explanations or asking follow-up questions if the user is confused or does not understand.

[1392] Step 13:

[1393] Users can answer questions posed by the AI, further deepening the discussion, and the AI ​​can provide additional information as needed.

[1394] Step 14:

[1395] The device translates the meeting content in real time, allowing participants who speak different languages ​​to understand the content.

[1396] Step 15:

[1397] After the meeting, the server uses the AI ​​to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1398] Step 16:

[1399] The generative AI summarizes the next action items discussed during the meeting, listing specific tasks such as "finish marketing materials by the next meeting."

[1400] Step 17:

[1401] The server notifies the user of the created minutes and a summary of the next action items, for example, via email or in-system notifications.

[1402] Step 18:

[1403] The emotion engine analyzes post-meeting emotional data and generates a report assessing meeting satisfaction and responsiveness.

[1404] Step 19:

[1405] Use the meeting information, minutes, and summary of next action items stored in the database to prepare for the next meeting, making it more effective.

[1406] Example 2

[1407] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1408] Conventional meeting support systems required the use of multiple tools for meeting preparation, progress, and follow-up, making efficient meeting management difficult. In particular, they were unable to grasp participants' emotions and reactions in real time and provide appropriate questions or supplementary information based on that information. Communication between participants who spoke different languages ​​also posed an obstacle to meeting management.

[1409] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1410] In this invention, the server includes means for inputting meeting schedules and agendas, means for receiving minutes and related materials from the previous meeting, means for saving the received information, means for inputting the minutes and agenda from the previous meeting to a generation AI and generating questions and supplementary information, means for notifying the start of the meeting, means for converting the meeting content into text in real time using speech recognition technology, means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information, means for the generation AI to provide real-time translation of the meeting content, means for recognizing user emotions using an emotion engine and for the generation AI to adjust the questions and supplementary information based on the recognized emotion data, means for the generation AI to create minutes and summarize next action items after the meeting ends, means for analyzing emotion data and generating a report evaluating satisfaction and responsiveness, means for notifying the created minutes and summaries of next action items, and means for using the saved data to prepare for the next meeting. This provides consistent support for meetings from preparation to progress and follow-up after the end, enabling appropriate responses based on user emotions, thereby improving the efficiency and quality of meetings.

[1411] "Scheduled Meeting" means a specific schedule of a meeting to be held on a specific date and time.

[1412] "Agenda" means the list of items or topics to be discussed at a meeting.

[1413] "Meeting minutes" refers to a document that records what was discussed and what decisions were made during a meeting.

[1414] "Related Materials" means any additional information or documents related to the agenda of the meeting.

[1415] "Generative AI" refers to a system that uses artificial intelligence technology to analyze data and generate questions and supplemental information.

[1416] "Voice recognition technology" refers to technology that converts voice into text data.

[1417] An "emotion engine" refers to a system that recognizes emotions by analyzing a user's facial expressions and tone of voice.

[1418] "Real-time translation" refers to technology that translates conversations into different languages ​​in real time.

[1419] "Meeting proceedings" refers to the procedures and processes for smoothly running a meeting.

[1420] "Next action items" refer to specific tasks or action items to be carried out before the next meeting.

[1421] "Emotional data" refers to data that indicates the emotional state of a user.

[1422] "Evaluation Report" means a report that evaluates the satisfaction and responsiveness of a meeting based on specific criteria.

[1423] The system of the present invention provides a series of functions that consistently support the preparation, progress, and follow-up of a conference. The system operates in cooperation with a server, terminals, and users. Each function of the system is implemented as follows:

[1424] 1. Schedule meetings and enter agendas

[1425] Users input meeting schedules and agendas into the system. This information is stored in a database by the server. For example, a user might input "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1426] 2. Uploading materials

[1427] Users upload the minutes of previous meetings and related documents (e.g., specifications for new products, marketing plans) to the system, and the server stores this information in a database.

[1428] 3. Input of meeting materials

[1429] The server retrieves the meeting agenda and related materials from the database and inputs them into the generative AI model. The generative AI model analyzes this data and generates important questions and supplemental information from the meeting content. For example, it generates a question such as, "Please tell me the status of software testing for the new product."

[1430] 4. Role assignment

[1431] The server assigns different roles to the generating AI, such as questioner, supplementary information provider, and minutes writer.

[1432] 5. Before the meeting begins

[1433] When a meeting starts, the user notifies the system that they want to start a meeting. The server then passes the prepared data to the generation AI, which prepares to support the progress of the meeting.

[1434] 6. Speech Recognition and Text Conversion

[1435] The device recognizes the audio of the meeting in real time and converts it into text format. This process records all utterances as text. For example, a utterance such as "When is the release date of the new product?" is sent to the server as text data.

[1436] 7. Real-time monitoring and questioning

[1437] The server uses generative AI to monitor the progress of the meeting in real time and provide appropriate questions and supplemental information. Based on the content of the meeting, the generative AI asks questions such as, "Please tell me about the status of software testing for our new product."

[1438] 8. Analysis by Emotion Engine

[1439] The device has a built-in emotion engine that analyzes the user's facial expressions and tone of voice to recognize their emotions. The recognized emotion data is sent to a server, and the generation AI adjusts the questions and supplementary information based on that data. For example, if the user is confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[1440] 9. Real-time translation

[1441] The device translates in real time during a meeting, ensuring that participants who speak different languages ​​can understand what is being said, for example translating what is being said in English into Japanese.

[1442] 10. Preparation of minutes

[1443] After the meeting, the server uses generative AI to create minutes based on the audio and text data from the meeting. For example, it might record, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1444] 11. Summary of next action items

[1445] The generative AI model then summarizes the next action items based on the discussion, such as a list of specific tasks such as "finish marketing materials by the next meeting."

[1446] 12. Emotion Data Analysis and Report Generation

[1447] After the meeting, the server analyzes the emotional data and generates a report assessing satisfaction and responsiveness of the meeting.

[1448] 13. Notice and Retention

[1449] The created minutes and next action items are notified to the user by the server and stored in a database.

[1450] This allows the system to provide consistent support from meeting preparation to progress and follow-up after the meeting has ended, and enables appropriate responses based on the user's emotions, thereby improving the efficiency and quality of meetings.

[1451] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1452] Step 1:

[1453] The user enters the meeting schedule and agenda. The "meeting date and time," "location," and "topic" entered by the user are sent to the system. The server saves this information in a database. This allows the system to manage basic meeting information. Example input: "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[1454] Step 2:

[1455] Users upload the minutes of previous meetings and related materials to the system. Related materials provided by users, such as "minutes," "specifications," and "marketing plans," are sent to the system. The server receives these materials and stores them in a database. This provides background information for the meeting.

[1456] Step 3:

[1457] The server retrieves the saved meeting agenda and related materials from the database. The retrieved "agenda," "minutes," and "materials" are input into the generative AI model. The generative AI model analyzes this data and generates important questions and supplementary information from the meeting content. The generated questions are saved on the server. For example, it generates a question such as, "Please tell me the status of software testing for the new product."

[1458] Step 4:

[1459] The server assigns roles to the generating AI, such as "questioner," "supplementary information provider," and "minutes writer," which determines the role the generating AI will play during the meeting.

[1460] Step 5:

[1461] When a meeting starts, the user notifies the system that they want to start a meeting. The server receives this notification and the entire system switches to meeting mode. The server then supplies pre-prepared data to the generation AI and begins supporting the progress of the meeting.

[1462] Step 6:

[1463] The device recognizes the audio of the meeting in real time and converts it into text. The input audio data is converted into "text data" using the device's voice recognition technology. The converted text data is sent to the server and stored. For example, the audio "When is the release date for the new product?" is converted into text.

[1464] Step 7:

[1465] The server uses generative AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information. It analyzes what is being said in the meeting in real time and asks questions based on the content of the meeting. For example, it might ask, "Please tell me about the status of software testing for our new product."

[1466] Step 8:

[1467] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions. The analyzed "emotion data" is sent from the device to a server. The server passes the received emotion data to the generation AI, which then adjusts the questions and supplementary information based on the emotion data. For example, if the user seems confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[1468] Step 9:

[1469] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand the content. Based on the input "voice data" and "text data," "translated text" is generated and provided to other participants. For example, content spoken in English can be translated into Japanese.

[1470] Step 10:

[1471] After the meeting ends, the server uses a generation AI to create minutes based on the audio and text data from the meeting. The generation AI analyzes the input audio and text data and generates minutes text. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the release date for the new product will be January 15, 2024."

[1472] Step 11:

[1473] The generative AI model summarizes next action items based on the content of the discussion. It analyzes the "minutes text" as input and generates "next action items." For example, it lists specific tasks such as "Complete marketing materials by the next meeting."

[1474] Step 12:

[1475] After the meeting ends, the server analyzes the emotional data and generates a report evaluating the satisfaction and responsiveness of the meeting. Based on the input "emotional data," a "satisfaction evaluation report" and a "responsiveness evaluation report" are generated. This allows the quality of the meeting to be objectively evaluated.

[1476] Step 13:

[1477] The server notifies the user of the minutes and next action items, and stores them in a database, which can be used to prepare for the next meeting.

[1478] (Application example 2)

[1479] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1480] Current meeting management systems often require manual preparation, progress, and follow-up after a meeting, which is labor-intensive and time-consuming. They also face challenges in being unable to meet advanced requirements, such as conducting meetings in multiple languages ​​or recognizing participants' emotions. In particular, there is a lack of systems that can properly manage workers' emotions and stress in production line work environments. Therefore, there is a need for a system that can simultaneously improve meeting efficiency and consider the emotions of workers.

[1481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1482] In this invention, the server includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for saving the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting the meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; an emotion engine that recognizes participants' emotions during the meeting and analyzes the emotion data to adjust questions and supplementary information; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; and means for using the saved data to prepare for the next meeting. This provides comprehensive support for meetings from preparation to progress and follow-up after the meeting, and by utilizing emotion data, it is possible to appropriately manage worker reactions and stress, enabling effective meeting management and improving the working environment.

[1483] The "means for inputting meeting schedules and agendas" is a function that allows a user to input schedule information such as the date and time of a meeting and the agenda into the system.

[1484] "Means for receiving minutes of the previous meeting and related materials" is a function for uploading minutes created at the previous meeting and materials related to the meeting to the system.

[1485] "Means for storing received information" refers to a function for storing meeting information and materials entered or uploaded into the system in a database.

[1486] "Means of inputting the previous meeting minutes and agenda into the generating AI and generating questions and supplementary information" refers to a function that inputs the previous meeting minutes and agenda into the AI ​​and automatically generates important questions and supplementary information based on them.

[1487] The "means for notifying the start of a conference" is a function for notifying the system when a user starts a conference.

[1488] "Means for converting meeting content into text in real time using voice recognition technology" is a function that uses voice recognition technology to convert voices spoken during a meeting into text in real time.

[1489] "Means for generative AI to monitor the progress of the meeting in real time and provide appropriate questions or supplementary information" refers to a function that monitors the progress of the meeting in real time and allows the AI ​​to provide questions or supplementary information at the appropriate time.

[1490] "Means for providing real-time translation of meeting content using generative AI" refers to a function that uses AI to translate meeting content in real time so that meeting participants who speak different languages ​​can understand the content.

[1491] "Means for adjusting questions and supplementary information by analyzing emotional data, including an emotion engine that recognizes participants' emotions during the meeting" refers to a function that uses an emotion engine that analyzes participants' facial expressions and tone of voice to obtain emotional data during the meeting, and then uses the AI ​​to adjust questions and supplementary information based on that data.

[1492] "A means for the generation AI to create minutes after the meeting and summarize action items for the next meeting" is a function in which the AI ​​automatically creates minutes based on the meeting content after the meeting and summarizes action items for the next meeting.

[1493] The "means for notifying the user of the created minutes and a summary of the next action item" is a function for notifying the user of the created minutes and a summary of the next action item.

[1494] "Means for using saved data in preparation for the next meeting" is a function for utilizing past meeting data saved in the database in preparation for the next meeting.

[1495] The system for implementing this invention is configured to provide various functions through cooperation between a server, a terminal, and a user. This system includes programs for realizing the following functions based on the claims.

[1496] First, the user enters the meeting schedule and agenda, and uploads the minutes of the previous meeting and related materials to the system. This information is sent to the server via the device and stored in the server's database. The server then inputs the stored information into the generation AI, which automatically generates questions and supplementary information based on the meeting content.

[1497] At the start of a meeting, the user uses their device to announce "start meeting." The device then converts the meeting content into text in real time using voice recognition technology, and the generation AI provides real-time questions and supplementary information. Furthermore, during the meeting, an emotion engine recognizes the emotions of participants and sends this data to the server. The generation AI then adjusts the questions and supplementary information based on the emotion data, optimizing the progress of the meeting. The generation AI also provides real-time translation for multilingual meeting participants.

[1498] After the meeting ends, the server uses generative AI to create minutes based on the audio and text data of the meeting and summarizes the next action items. This information is notified to the user and saved in a database. The saved data can be used to prepare for the next meeting.

[1499] As a specific example, let's imagine a new product release planning meeting, and the flow would be as follows:

[1500] 1. A user schedules a new product launch meeting to be held in the conference room on October 1, 2023 at 10:00 AM.

[1501] 2. Upload previous meeting minutes, new product specifications, and marketing plans into the system.

[1502] 3. The server obtains this information and inputs it into a generative AI model to generate questions and supplementary information in advance.

[1503] 4. When the conference starts, the user tells the system to "start the conference."

[1504] 5. The device converts the audio of the meeting into text, and the generative AI provides appropriate questions and follow-up comments.

[1505] 6. The device's built-in emotion engine analyzes the user's facial expressions and tone of voice and sends the emotion data to the server.

[1506] 7. Based on the emotion data, the server will ask questions and provide further explanations. If the user is confused, the AI ​​will ask questions such as, "Could you explain more specifically?"

[1507] 8. The device will translate the meeting content in real time, enabling multilingual support.

[1508] 9. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[1509] 10. The generated minutes and next action items are notified to the user and saved in the database.

[1510] The hardware used includes smartphones and head-mounted displays, and the software used includes the speech_recognition library (speech recognition) and the transformers library (generative AI and emotion recognition).

[1511] Example prompt sentence:

[1512] "A system that recognizes speech and analyzes emotions in real time during new product launch meetings"

[1513] "I want to automatically generate questions based on the presentation content and save them as meeting minutes."

[1514] In this way, the system improves meeting efficiency and the working environment.

[1515] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1516] Step 1:

[1517] The user inputs the meeting schedule and agenda. At this stage, the user inputs information such as "The new product launch meeting will be held on October 1, 2023 at 10:00" through the system interface. This obtains the meeting date and agenda as input data.

[1518] Step 2:

[1519] The user uploads the minutes of the previous meeting and related materials to the system. In this step, the user sends the minutes of the previous meeting, the specifications of the new product, the marketing plan, and other related materials to the system, which then receives these materials as input data.

[1520] Step 3:

[1521] The server stores the received information. The server accumulates the meeting schedule, agenda, and related materials obtained in steps 1 and 2 in a database. This stores the received information in the database and saves the accumulated data.

[1522] Step 4:

[1523] The server inputs the previous meeting minutes and agenda into the generation AI, which then generates questions and supplementary information. The server inputs the previous meeting minutes and agenda saved from the database into the generation AI model. The generation AI analyzes this data and automatically generates important questions and supplementary information related to the meeting content. The generated questions and supplementary information are obtained as output data.

[1524] Step 5:

[1525] The user notifies the system of the start of the conference. The user inputs "start conference" into the system via the terminal, and the conference starts. This acquires the conference start notification as input data.

[1526] Step 6:

[1527] The device uses speech recognition technology to convert the contents of the meeting into text in real time. The device picks up the meeting audio with its built-in microphone and converts the audio into text using the speech_recognition library. At this stage, the real-time converted text becomes the output data.

[1528] Step 7:

[1529] The server uses generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information. The server monitors the text data of the meeting content generated in step 6 and adds automatically generated questions and supplementary information using generation AI at the appropriate time. As a result, the questions and supplementary information provided become the output.

[1530] Step 8:

[1531] The device uses AI generation to provide real-time translation of the meeting content. The device translates text data in real time to support multiple languages, allowing participants who speak multiple languages ​​to understand the content. At this stage, the real-time translated meeting content is obtained as the output.

[1532] Step 9:

[1533] The device uses an emotion engine to recognize participants' emotions during the meeting and analyzes the emotion data to adjust questions and follow-up information. The device's built-in camera and microphone analyze facial expressions and vocal tone to recognize emotions. Based on this analysis, the emotion data is sent to a server, which then uses generative AI to adjust questions and follow-up information. This allows questions and follow-up information to be output that are adapted to the emotions.

[1534] Step 10:

[1535] The server uses the generation AI to create meeting minutes and summarize action items for the next meeting. After the meeting ends, the server uses the meeting content text data from step 6 and the emotion data from step 9 to automatically generate meeting minutes using the generation AI. In addition, based on the matters discussed during the meeting, the server summarizes action items for the next meeting. This outputs specific action items for the next meeting.

[1536] Step 11:

[1537] The server notifies the user of the created minutes and next action items. The server notifies the user of the automatically generated minutes and next action items and also stores them in a database. The final minutes and action items are notified to the user and are also stored as preparation data for the next meeting.

[1538] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1539] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1540] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1541] [Fourth embodiment]

[1542] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1543] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1544] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1545] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1546] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1547] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1548] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1549] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1550] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1551] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1552] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1553] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1554] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1555] The system of this invention provides a series of functions for enhancing conferences through collaboration between the server, terminals, and users. Below, we will explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[1556] overview

[1557] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for questions and supplementary information, real-time translation, meeting minutes, and a summary of upcoming action items.

[1558] System execution example

[1559] 1. Preparation Phase

[1560] The user inputs the meeting schedule and specific agenda into the system. For example, the user might input "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1561] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[1562] This information is stored in a database by the server.

[1563] 2. Before the meeting begins

[1564] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into the generative AI model.

[1565] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[1566] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[1567] 3. During a meeting

[1568] When a meeting begins, the user tells the system to "start the meeting."

[1569] The device recognizes the audio of the meeting in real time and converts it into text.

[1570] The server monitors the progress of the meeting in real time with a generative AI. For example, if a new product software is being discussed, the generative AI will ask appropriate questions, such as, "Could you please tell me more about the status of testing of the new feature?"

[1571] The user's statements are recorded, and the generating AI provides supplementary information as needed.

[1572] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[1573] 4. After the meeting

[1574] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1575] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[1576] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[1577] Specific examples

[1578] For example, if we take a new product release planning meeting, the flow would be as follows:

[1579] 1. A user schedules a new product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00.

[1580] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "marketing plan" to the system.

[1581] 3. The server obtains this information and inputs it into the generative AI model, which generates questions and supplementary information in advance.

[1582] 4. The conference begins and the user notifies the system to "start the conference."

[1583] 5. The device converts the audio of the meeting into text, and the server uses generated AI to provide appropriate questions and additional information.

[1584] 6. The device performs real-time translation, enabling multilingual support.

[1585] 7. After the meeting, the server uses AI to create minutes and summarize next action items.

[1586] 8. The server notifies the user of the minutes and next action items and stores them in the database.

[1587] In this way, the AI-Meeting Assistant system provides comprehensive support from meeting preparation to progress and follow-up after the meeting has ended, helping to ensure fruitful meetings.

[1588] The processing flow will be explained below.

[1589] Step 1:

[1590] A user logs in to the system and enters the schedule and agenda for a meeting. For example, the user enters, "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1591] Step 2:

[1592] A user uploads the minutes of a previous meeting and related documents to the system, such as a specification document for a new product or a marketing plan.

[1593] Step 3:

[1594] The server receives the meeting schedule, agenda, and related materials sent by the user, and then categorizes and stores the information in a database.

[1595] Step 4:

[1596] The server retrieves the saved meeting agenda, previous meeting minutes, and related materials from the database and inputs them into the generative AI model.

[1597] Step 5:

[1598] The generative AI analyzes the meeting content and generates important questions and supplementary information in advance, which is then stored on the server.

[1599] Step 6:

[1600] The server assigns roles to the generation AI, such as "questioner," "supplementary information provider," and "minutes creator."

[1601] Step 7:

[1602] When it is time for the meeting to start, the user notifies the system that the meeting should begin, for example by clicking a button.

[1603] Step 8:

[1604] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[1605] Step 9:

[1606] The server monitors the progress of the meeting in real time through a generative AI, which then asks appropriate questions to participants, such as, "Please tell me about the status of testing the software for our new product."

[1607] Step 10:

[1608] Users can answer questions posed by the AI, further deepening the discussion, and the AI ​​can provide additional information as needed.

[1609] Step 11:

[1610] The device translates the meeting content in real time, allowing participants who speak different languages ​​to understand the content.

[1611] Step 12:

[1612] After the meeting, the server uses the AI ​​to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1613] Step 13:

[1614] The generative AI summarizes the next action items discussed during the meeting, listing specific tasks such as "finish marketing materials by the next meeting."

[1615] Step 14:

[1616] The server notifies the user of the created minutes and a summary of the next action items, for example, via email or in-system notifications.

[1617] Step 15:

[1618] Use the meeting information, minutes, and summary of next action items stored in the database to prepare for the next meeting, making it more effective.

[1619] Example 1

[1620] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1621] In conventional conferencing systems, meeting preparation, proceeding, and minutes creation are done manually, which requires time and effort. Furthermore, difficulties with real-time speech recognition and translation, and the provision of appropriate questions and supplementary information, result in a decline in meeting efficiency and quality. This invention aims to solve these problems by automating meeting preparation, proceeding, minutes creation, and summarizing next action items.

[1622] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1623] In this invention, the server includes means for inputting meeting schedules and agendas, means for receiving minutes and related materials from the previous meeting, means for storing the received information, means for inputting the minutes and agenda from the previous meeting into a generative AI model and generating questions and supplementary information, means for notifying the start of the meeting, means for converting the meeting content into text in real time using speech recognition technology, means for the generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information, means for the generative AI model to provide real-time translation of the meeting content, means for the generative AI model to create minutes and summarize next action items after the meeting ends, means for notifying the created minutes and summaries of next action items, and means for using the stored data to prepare for the next meeting. This makes it possible to streamline a series of meeting processes, such as preparing for the meeting, conducting the meeting, creating minutes, and summarizing next action items, thereby improving the quality and productivity of meetings.

[1624] A "meeting schedule and agenda entry means" is an interface through which a user enters meeting dates, times, locations, and agendas into the system.

[1625] The "means for receiving minutes of previous meetings and related materials" is an interface that allows users to upload records of previous meetings and documents related to meetings to the system.

[1626] The "means for storing received information" is a function for storing the meeting schedule and agenda entered by the user and the uploaded materials in a database.

[1627] "Means of inputting previous meeting minutes and agendas into a generative AI model to generate questions and follow-up information" refers to the process and techniques for inputting stored meeting data into a generative AI model to generate important questions and follow-up information.

[1628] The "means for notifying the start of a conference" is an operation interface that allows a user to notify the system of the start of a conference.

[1629] "Means for converting meeting content into text in real time using speech recognition technology" refers to technology and devices for converting speech made during a meeting from voice to text in real time.

[1630] "Means for a generative AI model to monitor the progress of a meeting in real time and provide appropriate questions and supplementary information" refers to a technology that enables a generative AI model to analyze the content of a meeting, ask appropriate questions on the spot, and provide supplementary information.

[1631] "Means for providing real-time translation of meeting content using generative AI models" refers to technologies and devices that translate speech in real time during a meeting so that all participants who speak different languages ​​can understand the content.

[1632] "Means for a generative AI model to create minutes after a meeting and summarize action items for the next meeting" refers to a technology that analyzes the contents of a meeting from recorded audio and text data, and generates minutes and action items to be addressed at the next meeting.

[1633] The "means for notifying the user of the created minutes and a summary of the next action item" is a function for notifying the user of the created minutes and action items.

[1634] The "means for utilizing stored data in preparation for the next meeting" refers to the process and technology for utilizing stored meeting data in preparation for the next meeting.

[1635] The system of the present invention provides a series of functions for efficiently proceeding and managing a conference through collaboration between a server, terminals, and users. An embodiment of the program of this system will be described in detail below.

[1636] System Configuration

[1637] server

[1638] The server plays a central role in the system and provides the following functions:

[1639] 1. Database management: Store, retrieve, and manage meeting schedules, agendas, minutes, related materials, etc. MySQL or PostgreSQL is used as the RDBMS.

[1640] 2. Generative AI model interface: Input data into a generative AI model (e.g., OpenAI GPT-4) and have it generate questions and supplemental information. The generative AI model operates by receiving the following prompts:

[1641] "Generate important questions and supporting information to discuss at the next meeting."

[1642] 3. Real-time data processing: Audio and text data from meetings are processed in real time and passed to the generative AI model.

[1643] Terminal

[1644] The device is used at the meeting site and provides the following features:

[1645] 1. Speech Recognition: Converts meeting audio into text in real time using Google Cloud Speech-to-Text and Azure Speech Services.

[1646] 2. Real-time translation: Translates meeting content in real time, enabling multilingual support. Translation is performed using the Google Translate API and DeepL API.

[1647] 3. User operation interface: Provides a UI for users to start meetings and upload materials, using web forms or dedicated apps.

[1648] User

[1649] Users operate the system and fulfill the following roles:

[1650] 1. Entering meeting schedules and agendas: Enter the meeting date, time, and agenda into the system. For example, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[1651] 2. Upload related materials: Upload the minutes of the previous meeting and related materials using drag-and-drop or file selection dialogs.

[1652] 3. Conference start notification: Notify the system that a conference is about to begin.

[1653] Specific examples

[1654] Take the example of a new product release planning meeting:

[1655] 1. Preparation Phase

[1656] The user enters "New product launch meeting to be held in conference room C at 10:00 on October 1, 2023" into the web form and sets up the meeting schedule in the system.

[1657] Users upload documents such as "minutes of the previous meeting," "specifications for a new product," and "marketing plans" into the system.

[1658] The server stores this information in a database.

[1659] 2. Before the meeting begins

[1660] The server retrieves the meeting agenda and related materials from the database and inputs them into the generative AI model.

[1661] The generative AI model generates key questions and supplementary information based on the prompt sentence and stores it on the server.

[1662] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generative AI model.

[1663] 3. During a meeting

[1664] When a meeting begins, the user tells the system to "start the meeting."

[1665] The device converts the meeting audio into text in real time and sends it to a generative AI model.

[1666] The server uses a generative AI model to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information.

[1667] The device provides real-time translation, allowing all participants, even those speaking different languages, to understand the content of the meeting.

[1668] 4. After the meeting

[1669] The server creates meeting minutes based on the voice and text data recorded by the generative AI model. For example, it might record, "At the meeting held on October 1, 2023, the release date for the new product was decided to be January 15, 2024."

[1670] The generative AI model summarizes action items for the next meeting based on the discussion, listing specific tasks such as "finish marketing materials by the next meeting."

[1671] The server notifies the user of the created minutes and next action items and stores them in a database.

[1672] In this way, the form for implementing the invention improves the quality and productivity of meetings by coordinating the server, terminals, and users to consistently improve the efficiency of meeting preparation, progress, and follow-up after the meeting has ended.

[1673] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1674] Step 1:

[1675] Users enter meeting schedules and agendas

[1676] Input: The user enters the date, time, location, and agenda of the meeting into a web form or dedicated app. For example, they might enter, "New product launch meeting to be held in Conference Room C on October 1, 2023 at 10:00."

[1677] Data processing: The input information is sent to the server as text data. The meeting ID, date, time, location, and agenda are parsed and formatted as structured data.

[1678] Output: The formatted data is stored in an internal database.

[1679] Specific behavior: A front-end component (e.g., HTML form, JavaScript) is executed to receive form input and a POST request is executed to send the input to the server.

[1680] Step 2:

[1681] Users upload previous meeting minutes and related materials

[1682] Input: Users upload files such as minutes from a previous meeting, specifications for a new product, or a marketing plan to the system using drag-and-drop operations or file selection dialogs.

[1683] Data processing: Uploaded files are received by the server and stored in a location appropriate for the file format. Metadata (file name, upload date, etc.) is also generated.

[1684] Output: The file body and metadata are saved to the database and file storage.

[1685] Specific behavior: A file selection dialog on the front end, a file upload API endpoint, and a storage system for saving files (e.g. AWS S3, Google Cloud Storage) are used.

[1686] Step 3:

[1687] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database.

[1688] Input: Before a meeting starts, the system queries the database based on the meeting ID entered by the user.

[1689] Data processing: The necessary data (agenda, minutes, related materials) is retrieved from the database and formatted to be input into the generative AI model.

[1690] Output: The formatted data is temporarily stored in memory.

[1691] Specific operations: Executes SQL queries, formats and transforms data, and caches retrieved data in memory.

[1692] Step 4:

[1693] Input the previous meeting minutes and agenda into the generative AI model to generate questions and supplemental information

[1694] Input: Formatted minutes and agenda are sent to the generative AI model with the prompt, "Generate key questions and supporting information to discuss at the next meeting."

[1695] Data Computation: Generative AI models analyze text data and generate questions and follow-up information based on specified prompts.

[1696] Output: The generated questions and supplementary information are returned in text format to the server and stored in an appropriate database.

[1697] Specific operation: An API request is sent to the generative AI model, and the generated content is saved.

[1698] Step 5:

[1699] User tells the system to "start a meeting"

[1700] Input: To start a meeting, the user clicks the "Start Meeting" button on the dedicated app or web interface.

[1701] Data processing: The server notifies the entire system that the conference has started and performs the necessary initialization processing.

[1702] Output: The conference state is updated to "Started" and relevant system components are notified.

[1703] Specific operation: The button click event is processed, the conference state management system is updated, and each component is notified.

[1704] Step 6:

[1705] The device recognizes the audio of the meeting in real time and converts it into text.

[1706] Input: Voice data collected by the device's microphone during the meeting is sent to the voice recognition system.

[1707] Data processing: A speech recognition system (such as Google Cloud Speech-to-Text or Azure Speech Services) analyzes the voice data and converts it into text data in real time.

[1708] Output: The text of the remarks is sent to the server and saved as a meeting record.

[1709] Specific operations: Collects voice data, calls voice recognition API, and sends and saves text data.

[1710] Step 7:

[1711] Generative AI models monitor meeting progress in real time and provide relevant questions and follow-up information

[1712] Input: Real-time transcribed meeting content is fed into the generative AI model.

[1713] Data Computation: The generative AI model analyzes the supplied text data and generates appropriate questions and supplemental information.

[1714] Output: Generated questions and supplemental information are provided to meeting participants in real time.

[1715] Specific operation: Real-time data is supplied to the generative AI model, and the generated content is displayed on the interface.

[1716] Step 8:

[1717] The device performs real-time translation

[1718] Input: Text data output from the generative AI model and speech recognition system is sent to the real-time translation system.

[1719] Data calculation: A translation system (Google Translate API or DeepL API) translates the text data into the specified language.

[1720] Output: The translated text is provided to meeting participants in real time.

[1721] Specific operation: Text data is sent to the translation API and the translated text is displayed.

[1722] Step 9:

[1723] After the meeting, the server creates minutes based on the generative AI model and summarizes next action items.

[1724] Input: Recorded audio and text data is sent to the generative AI model.

[1725] Data Computation: Generative AI models analyze text data and summarize meeting minutes and next action items.

[1726] Output: The generated minutes and next action items are stored in a database.

[1727] Specific operations: Data is sent to the generative AI model, and meeting minutes and action items are automatically generated and saved.

[1728] Step 10:

[1729] The server notifies you of the minutes created and a summary of upcoming action items

[1730] Input: The generated meeting minutes and action items are retrieved from the database.

[1731] Data Processing: Notification content is formatted into a user-friendly format.

[1732] Output: The formatted notification content is sent to the user via email and / or in-app notification.

[1733] Specific operations: Retrieving data from the database, formatting the notification content, and sending the notification.

[1734] (Application example 1)

[1735] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1736] There is a demand for a system that can centrally manage meetings and briefings held within factories, including efficient progress, information sharing, and multilingual support. In addition, collaboration between robots and humans can improve productivity, and real-time speech recognition and translation, as well as the provision of appropriate questions and supplementary information, are necessary. However, conventional systems have difficulty in effectively performing these tasks, consuming a great deal of time and resources. This invention was devised to solve these problems and realize efficient and effective meeting progress.

[1737] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1738] In this invention, the server includes: means for inputting meeting schedules and agendas; means for receiving minutes and related materials from previous meetings; means for saving the received information; means for inputting the minutes and agenda to a generation AI and generating questions and supplementary information; means for notifying the start of a meeting; means for converting the meeting content into text in real time using voice recognition technology; means for the generation AI to monitor the progress of the meeting in real time and provide appropriate questions and supplementary information; means for the generation AI to provide real-time translation of the meeting content; means for the generation AI to create minutes and summarize next action items after the meeting; means for notifying the created minutes and summaries of next action items; means for using the saved data to prepare for the next meeting; means for streamlining meetings and briefings held in factories and supporting collaboration between humans and robots; and means for converting the meeting content into text in real time, providing translations, and for the generation AI to provide appropriate questions and supplementary information. This enables meetings to be conducted efficiently and effectively in factories, improving productivity through collaboration between robots and humans.

[1739] "Means for inputting meeting schedules and agendas" refers to a function for registering meeting dates and topics in the system in advance.

[1740] "Means for receiving minutes of previous meetings and related materials" is a function for uploading records of past meetings and necessary materials to the system.

[1741] "Means for storing received information" refers to a function for storing uploaded minutes and materials in a database.

[1742] "Means of inputting the previous minutes and agenda into the generating AI and generating questions and supplementary information" refers to a function that inputs saved minutes and agendas into the generating AI model and automatically generates questions and supplementary information required for the meeting.

[1743] The "means for notifying the start of the conference" is a function for notifying participants of the start time of the conference.

[1744] "Means of converting meeting content into text in real time using voice recognition technology" is a function that uses technology to convert statements made during a meeting into text data in real time.

[1745] "Means for the generative AI to monitor the progress of the meeting in real time and provide appropriate questions or supplementary information" refers to a function in which the generative AI monitors the progress of the meeting and instantly provides questions or supplementary information.

[1746] "Means for providing real-time translation of meeting content using generative AI" is a function that translates meeting content in real time to ensure smooth communication between participants who speak different languages.

[1747] "A means for the generation AI to create minutes after the meeting and summarize the next action items" is a function that automatically creates minutes based on the content of the meeting and summarizes the next action items to be taken.

[1748] "Means for notifying the created minutes and a summary of the next action item" is a function for notifying participants of the created minutes and the next task.

[1749] "Means of using saved data to prepare for the next meeting" is a function that helps prepare for the next meeting by referring to past meeting records and materials.

[1750] "A means to streamline meetings and briefings held within factories and support collaboration between humans and robots" is a function that helps robots participate in meetings and information sharing held within factories so that they can proceed efficiently.

[1751] "A means of converting meeting content into text in real time, providing translation, and having a generation AI provide appropriate questions and supplementary information" refers to a function that converts statements into text data in real time, translates them as needed, and then has a generation AI provide questions and supplementary information.

[1752] The system of the present invention improves the efficiency of meetings and briefings in factories and provides a series of functions to support collaboration between humans and robots. The specific configuration and operation of the system are described below.

[1753] System Overview

[1754] 1. Preparation Phase

[1755] Users can use smartphones, tablets, or even the terminals of robots in factories to enter meeting schedules and specific agendas. For example, they can enter information such as "October 1, 2023, 10:00 AM, in Conference Room C in the factory, to hold a meeting to launch a new product line."

[1756] Users upload previous meeting minutes and related materials (e.g., production reports, safety guidelines) into the system.

[1757] This information is stored in a database by the server.

[1758] 2. Before the meeting begins

[1759] The server retrieves the meeting agenda, previous meeting minutes, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT).

[1760] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[1761] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[1762] 3. During a meeting

[1763] The user tells the system to "start a meeting."

[1764] The terminals (smartphones, factory robots) recognize the audio of the meeting in real time and convert it into text using Google Cloud Speech-to-Text.

[1765] The server monitors the progress of the meeting in real time with a generative AI. For example, when a new software product is discussed, the generative AI asks appropriate questions such as, "Please tell me more about the status of testing of the new feature."

[1766] The device uses the Google Translate API to provide real-time translation in multiple languages, ensuring that participants who speak different languages ​​can understand the content.

[1767] 4. After the meeting

[1768] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it could record that "at the meeting held on October 1, 2023, it was decided that the release date for a new product would be January 15, 2024."

[1769] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[1770] The created minutes and next action items are notified to the user from the server and are further stored in a database.

[1771] Examples of specific examples and prompts

[1772] For example, if we take a new product release planning meeting, the flow would be as follows:

[1773] 1. A user schedules a meeting to launch a new product line in conference room C in the factory at 10:00 AM on October 1, 2023.

[1774] 2. The user uploads the "minutes of the previous meeting," "specifications for the new product," and "safety guidelines" to the system.

[1775] 3. The server acquires this information and inputs it into a generative AI model, which generates questions and supplementary information in advance.

[1776] 4. The conference begins and the user tells the system to "start the conference."

[1777] 5. The device converts the meeting audio into text using Google Cloud Speech-to-Text, and the server provides appropriate questions and follow-up questions using generated AI.

[1778] 6. The device uses the Google Translate API for real-time translation, enabling multilingual support.

[1779] 7. After the meeting, the server uses the AI ​​to create minutes and summarize the next action items.

[1780] 8. The server notifies the user of the minutes and next action items and saves them in the database.

[1781] Prompt Sentence Examples

[1782] User: "We will hold a launch meeting for the new product line in conference room C in the factory on October 1, 2023 at 10:00 AM."

[1783] Server: "Please upload the last meeting minutes, new product specifications, and safety guidelines."

[1784] User: "I uploaded it."

[1785] Server: "Do you want to start the meeting?"

[1786] User: "Yes, let's start."

[1787] Server (generative AI): "Tell me more about the challenges you're facing with your current product line."

[1788] Server (generative AI): "Please tell me the status of the commissioning of the new product line."

[1789] Server: "The meeting has ended. The minutes are being compiled..."

[1790] Server: "I've summarized our next action items: 1. Report test results for new product line. 2. Review safety guidelines."

[1791] In this way, a system is provided that enables meetings within a factory to be conducted efficiently and enables collaboration between robots and humans.

[1792] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1793] Step 1:

[1794] Users use smartphones, tablets, or robotic terminals in factories to enter meeting schedules and agendas. The entered data includes the date, time, location, and topic of the meeting. The entered information is sent to a server and stored in a database. The meeting is then prepared.

[1795] Input: Meeting schedule and agenda

[1796] Output: Meeting schedules and agendas stored in a database

[1797] Step 2:

[1798] Users upload minutes of previous meetings and related materials, such as production reports and safety guidelines, to the system, which the server then stores in a database.

[1799] Input: Minutes of previous meetings and related materials

[1800] Output: Meeting minutes and related materials stored in a database

[1801] Step 3:

[1802] The server retrieves the meeting agenda, minutes of the previous meeting, and related materials from the database and inputs this information into a generative AI model (e.g., OpenAI GPT). The generative AI automatically generates important questions and supplementary information from the meeting content and saves the output on the server.

[1803] Input: Agendas, minutes, and related materials retrieved from the database

[1804] Output: Auto-generated questions and additional information

[1805] Step 4:

[1806] When a conference starts, the user notifies the system that the conference is about to begin. The server then notifies all participants that the conference has begun.

[1807] Input: Meeting start notification

[1808] Output: Notification to participants

[1809] Step 5:

[1810] Devices (smartphones, factory robots) recognize speech during meetings in real time and convert it into text using Google Cloud Speech-to-Text. The converted speech is sent to a server, where a generating AI monitors the progress of the meeting in real time.

[1811] Input: Audio data during the meeting

[1812] Output: Real-time textualized speech data

[1813] Step 6:

[1814] The server analyzes the text of the meeting, which has been converted by the generative AI, and provides appropriate questions and supplementary information. For example, if a new product software is being discussed, the generative AI might ask, "Please tell me more about the status of testing of the new function."

[1815] Input: Translated meeting contents

[1816] Output: Questions and supplementary information generated by the generative AI

[1817] Step 7:

[1818] The device uses the Google Translate API to translate meeting content in real time and display it in a format that is easy to understand for participants who speak different languages, thereby enabling multilingual support.

[1819] Input: Translated meeting contents

[1820] Output: Real-time translated meeting content

[1821] Step 8:

[1822] After the meeting, the server uses generative AI to create minutes based on the audio and text data of the meeting. For example, the minutes might read, "At the meeting on October 1, 2023, it was decided that the new product release date would be January 15, 2024."

[1823] Input: Meeting audio and text data

[1824] Output: Generated meeting minutes

[1825] Step 9:

[1826] Based on the discussion, the generative AI summarizes action items for the next meeting, such as listing specific tasks like "complete marketing materials by the next meeting." This provides a concrete guide to action.

[1827] Input: Minutes and meeting contents

[1828] Output: List of next action items

[1829] Step 10:

[1830] The server notifies the user of the created minutes and next action items and stores them in a database, allowing the user to check the minutes and next action items, thereby streamlining preparations for the next meeting.

[1831] Input: Generated meeting minutes and action items

[1832] Output: Notify user and save to database

[1833] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1834] The system of this invention provides a series of functions for enhancing conferences through collaboration between servers, terminals, and users. Furthermore, by combining it with an emotion engine that recognizes users' emotions, more effective conference management is achieved. Below, we explain the program processing of this system in natural language and show how the system is implemented using concrete examples.

[1835] overview

[1836] The system offers a range of features to assist with meeting preparation, conduct, and post-processing, including meeting scheduling, document uploading, real-time speech recognition, generative AI for asking questions and providing additional information, real-time translation, creating meeting minutes, summarizing upcoming action items, and recognizing user emotions with an emotion engine.

[1837] System execution example

[1838] 1. Preparation Phase

[1839] The user inputs the meeting schedule and specific agenda into the system. For example, the user inputs "New product launch meeting to be held in conference room C on October 1, 2023 at 10:00."

[1840] Users upload the minutes of previous meetings and related materials (e.g., specifications for new products, marketing plans) into the system.

[1841] This information is stored in a database by the server.

[1842] 2. Before the meeting begins

[1843] The server retrieves the saved meeting agenda, previous minutes, and related materials from the database and inputs this information into the generative AI model.

[1844] The generative AI generates important questions and supplementary information from the meeting content in advance and stores them on the server.

[1845] The server assigns roles such as "questioner," "supplementary information provider," and "minutes writer" to the generating AI.

[1846] 3. During a meeting

[1847] When a meeting begins, the user tells the system to "start the meeting."

[1848] The device recognizes the audio of the meeting in real time and converts it into text, so that all statements are recorded as text.

[1849] The server monitors the progress of the meeting in real time through generative AI and asks appropriate questions, such as, "Please tell me the status of software testing for the new product."

[1850] The device's built-in emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotions.

[1851] The recognized emotion data is sent to a server, and the generation AI adjusts the questions and supplementary information based on that information. For example, if the user appears confused, the generation AI will ask a follow-up question such as, "Could you explain in more detail?"

[1852] The device translates in real time during a meeting, ensuring that all participants, even those speaking different languages, can understand what is being said.

[1853] 4. After the meeting

[1854] The server uses a generative AI to create meeting minutes based on the audio and text data of the meeting. For example, it might record something like, "At the meeting on October 1, 2023, it was decided that the release date for the new product will be January 15, 2024."

[1855] The generative AI summarizes action items for the next meeting based on the discussion, for example, listing specific tasks such as "Complete marketing materials by the next meeting."

[1856] Analyzes emotional data after the meeting and generates a report evaluating satisfaction and responsiveness of the meeting.

[1857] The created minutes and next action items are notified to the user from the server and are further stor...

Claims

1. a means for entering meeting schedules and agendas; A means of receiving previous meeting minutes and related materials; a means for storing the received information; and A means to input the previous minutes and agenda into the generation AI and generate questions and supplementary information; a means for announcing the start of a meeting; A means of converting meeting content into text in real time using voice recognition technology, A means for generative AI to monitor meeting progress in real time and provide appropriate questions and follow-ups; A means to provide real-time translation of meeting content using generative AI, After the meeting, the AI ​​will create minutes and summarize action items for the next meeting. A means of communicating the minutes prepared and a summary of upcoming action items; and means for utilizing the stored data in preparation for the next meeting.

2. The system according to claim 1 , further comprising means for the generation AI to pre-set a plurality of roles to participate in the conference.

3. The system of claim 1 , wherein the generative AI further comprises means for automatically formatting meeting minutes and next action item summaries.

Images