system

Abstract

We provide the system. [Solution] In online meetings, the means of receiving meeting requests, A method for obtaining the calendar information of meeting participants and calculating the optimal date and time, A means of collecting and sharing materials and data related to the meeting, A method for managing the scheduled time for each agenda item during a meeting and presenting relevant data in real time, A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes, A method for automatically extracting action plans from meeting minutes and assigning them to the responsible person, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In current online meeting systems, the processes from meeting preparation to progress and follow-up are fragmented, and there are often many inconveniences and time-consuming tasks. In particular, there are a wide variety of tasks such as meeting requests, participant coordination, collection of relevant materials, timekeeping during meetings, creation of meeting minutes, and management of action plans, making it difficult to perform these tasks efficiently and integratively. As a result, there is a risk of a decline in meeting quality and productivity. Therefore, there is a need for a system that solves this problem and improves and automates the overall process.

Means for Solving the Problems

[0005] The present invention solves the above-mentioned problems by providing a system for online meetings that includes the following means: means for receiving meeting requests; means for obtaining calendar information of meeting participants and calculating the optimal date and time; means for collecting and sharing materials and data related to the meeting; means for managing scheduled time for each agenda item during the meeting and presenting relevant data in real time; means for recording the meeting content, automatically transcribing it into text, and generating meeting minutes; and means for automatically extracting action plans from the meeting minutes and assigning them to the responsible parties. Furthermore, by including means for automatically sending reminder notifications to meeting participants and means for managing and reporting the progress of action plans, the entire meeting process can be made more efficient and automated, thereby improving productivity.

[0006] An "online meeting" is a type of meeting where participants from multiple locations gather together via the internet to discuss and share information in real time.

[0007] "Means of receiving requests" refers to the function of inputting and sending information such as meeting requests, participant lists, and agenda items into the system.

[0008] "Means of obtaining calendar information" refers to a function that checks participants' schedules and reads their free time and other appointments through the system.

[0009] The "method for calculating the optimal date and time" is a function that analyzes the calendar information of all participants and automatically determines the date and time when the most people can attend.

[0010] "Means for collecting and sharing materials and data" refers to the function of gathering information and documents related to the meeting agenda and distributing them to all participants in advance.

[0011] "A means of managing the scheduled time for each agenda item" refers to a function that measures and manages the time allocated to each agenda item during a meeting.

[0012] "Means of presenting relevant data in real time" refers to a function that instantly displays relevant information and materials in response to the moment of discussion.

[0013] "A method for recording meeting content, automatically converting it to text, and generating meeting minutes" refers to a function that records audio during a meeting, converts it to text, and then formats and distributes it as meeting minutes after the meeting has ended.

[0014] The "means for automatically extracting action plans and assigning them to responsible personnel" refer to a function that analyzes and extracts specific action instructions from meeting minutes and then notifies and assigns them to the appropriate members.

[0015] "Automatic reminder notification" refers to a function that automatically sends notifications to participants before the start time of a meeting or before important agenda items.

[0016] "Means for managing and reporting the progress of action plans" refers to the function of tracking the progress of assigned action instructions and reporting them when necessary. [Brief explanation of the drawing]

[0017] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing apparatus and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

MODE FOR CARRYING OUT THE INVENTION

[0018] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the language used in the following description will be explained.

[0020] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0021] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0022] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] The system of this invention combines multiple functions to enable the efficient and automated operation of online meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[0039] 1. Accepting requests

[0040] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0041] The server receives this request and saves it to the database. This data is used for subsequent processing.

[0042] 2. Scheduling

[0043] The server communicates with each user's calendar API to retrieve the calendar information of all participants.

[0044] Based on the acquired calendar information, the server calculates the optimal date and time. For example, it lists dates and times when many participants are available.

[0045] These proposed dates and times will be notified to the users for each participant to review. The date and time will be finalized upon final approval.

[0046] 3. Meeting preparations

[0047] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0048] The server organizes the collected data and uploads it to a shared folder.

[0049] Furthermore, the server creates meeting minutes templates based on the agenda items and distributes them to all participants.

[0050] 4. Meeting Management

[0051] The server sends a reminder notification to the user at the start of the meeting.

[0052] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data to the user's device in real time, depending on the agenda item.

[0053] 5. Recording of discussions and creation of meeting minutes.

[0054] The device records the meeting content and sends the data to the server.

[0055] The server converts the audio data into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[0056] 6. Action plan generation and management

[0057] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the responsible person.

[0058] Users manage the progress of their assigned action plans and update them through the system as needed.

[0059] The server will periodically report the progress of the action plan to all participants.

[0060] Specific example

[0061] For example, suppose a company wants to hold an online meeting for a new product development project. In this case, the user proposes the meeting regarding the new product development, enters the date and time, participant list, and agenda into the system, and submits the request.

[0062] The server receives the request and retrieves each participant's calendar information. It calculates the optimal date and time and notifies everyone. All participants review the proposed dates and times and finalize the date and time.

[0063] Before the meeting, terminals collect market research data and competitor analysis materials, and the server organizes and uploads them to a shared folder. Additionally, meeting minute templates tailored to the agenda are generated and distributed to participants in advance.

[0064] During the meeting, the server keeps track of the time and displays relevant data in real time for each agenda item. The meeting content is recorded by the terminals, and the server converts the audio data into text to automatically create meeting minutes.

[0065] After the meeting ends, the server extracts action plans from the meeting minutes and assigns them to the responsible parties. Users manage and update the progress of the action plans within the system, and the server reports the progress to all participants.

[0066] As a result, it becomes possible to streamline and automate the entire process, from meeting preparation to conducting and follow-up.

[0067] The following describes the processing flow.

[0068] Step 1:

[0069] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0070] Step 2:

[0071] The server receives this request and saves it to the database. This data is used for subsequent processing.

[0072] Step 3:

[0073] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[0074] Step 4:

[0075] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[0076] Step 5:

[0077] The server notifies users of a list of possible dates and times, and each participant confirms them. Based on the participants' confirmations, the optimal date and time are finalized.

[0078] Step 6:

[0079] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0080] Step 7:

[0081] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[0082] Step 8:

[0083] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[0084] Step 9:

[0085] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[0086] Step 10:

[0087] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[0088] Step 11:

[0089] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[0090] Step 12:

[0091] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[0092] Step 13:

[0093] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[0094] Step 14:

[0095] The server automatically generates meeting minutes based on the generated text. The minutes are organized according to a template distributed in advance.

[0096] Step 15:

[0097] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[0098] Step 16:

[0099] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[0100] Step 17:

[0101] Manage the progress of action plans assigned to users. Update task progress on the system.

[0102] Step 18:

[0103] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[0104] The above is the specific processing flow of the program for this system.

[0105] (Example 1)

[0106] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0107] Current online meeting systems make it difficult to efficiently manage the entire process, from meeting preparation to execution and follow-up. Furthermore, tasks such as timekeeping, real-time presentation of relevant materials during meetings, and post-meeting minute-taking and action plan extraction and management are not automated, relying heavily on manual processes. This results in significant effort and time being required for effective meeting management.

[0108] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0109] In this invention, the server includes means for receiving meeting requests, means for obtaining the schedule information of meeting participants and calculating the optimal date and time, means for collecting and sharing materials and information related to the meeting, means for managing the scheduled time for each agenda item during the meeting and presenting relevant information in real time, means for recording the meeting content, automatically transcribing it into text to generate meeting minutes, means for automatically extracting action plans from the meeting minutes and assigning them to responsible persons, means for sending reminders at the start of the meeting, means for sending meeting recording data to the server, and means for periodically reporting the progress of the action plans. This makes it possible to streamline and automate the entire process from meeting preparation to execution and follow-up.

[0110] "Means for receiving meeting requests" refers to the means by which users propose online meetings and send detailed information such as the date and time, participant list, and agenda to the server.

[0111] "A means of obtaining the schedule information of meeting participants and calculating the optimal date and time" refers to a method in which the server communicates with each participant's calendar API and calculates the optimal date and time that all participants can attend based on the information obtained.

[0112] "Means for collecting and sharing materials and information related to a meeting" refers to a method by which terminals collect necessary materials for a meeting from shared folders or external databases, and a server organizes and shares them with participants.

[0113] "A means of managing scheduled time for each agenda item during a meeting and presenting relevant information in real time" refers to a method in which a server manages the scheduled time during the meeting and presents relevant materials and information in real time as needed.

[0114] "A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes" refers to a method in which a terminal records the audio during a meeting, and a server uses speech recognition technology to transcribe that data into text and compile it into meeting minutes.

[0115] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a method in which a server analyzes the text of the meeting minutes, extracts specific action items, and automatically assigns them to the appropriate personnel.

[0116] "Method for sending reminders at the start of a meeting" refers to a method by which the server sends a reminder to all participants to notify them of the start time as the meeting is about to begin.

[0117] "Means for sending meeting recording data to the server" refers to the means by which a terminal uploads the recorded meeting audio data to the server.

[0118] "Means for periodically reporting on the progress of the action plan" refers to a method by which the server periodically checks the implementation status of the extracted action plan and reports its progress to all participants.

[0119] The system of the present invention enables the efficient and automated operation of online meetings through the collaborative roles of the server, terminal, and user. Specific embodiments are described below.

[0120] Accepting requests

[0121] A user enters and submits an online meeting request. This includes details such as the date and time, participant list, and agenda. For example, suppose a user submits a request for "Meeting on October 15, 2023 at 2:00 PM, with participants A, B, and C, on the topic of new product development." The server receives this request and stores it in the database. This makes the meeting details available for subsequent processing.

[0122] Example prompt:

[0123] I would like to hold a meeting regarding new product development on October 15, 2023, at 2:00 PM. The participants are Person A, Person B, and Person C. The topic of discussion will be "New Product Development".

[0124] Schedule adjustment

[0125] The server accesses the calendar APIs of all participants (e.g., Google® Calendar API, Outlook Calendar API) to retrieve calendar information. Based on this information, the server calculates the optimal date and time and notifies the users of the suggested dates and times. Each user who receives the notification reviews the suggested dates and times and accepts the optimal one. Through this confirmation, the final date and time are determined.

[0126] Example prompt:

[0127] Please obtain each participant's calendar information, calculate a date and time that works for everyone, and notify them accordingly.

[0128] Meeting preparations

[0129] The terminal collects materials and data related to the meeting. These are collected from shared folders (e.g., Google Drive, Dropbox) or external databases (e.g., a company's internal database). The server organizes these materials and uploads them to the shared folder, as well as generating meeting minutes templates based on the agenda and distributing them to participants in advance.

[0130] Example prompt:

[0131] Collect market research data and competitive analysis materials, upload them to a shared folder, and generate a meeting minutes template to distribute to participants.

[0132] Meeting progress

[0133] The server sends a reminder notification at the start of the meeting. During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data related to the agenda item on the user's device in real time. For example, during a discussion on new product development, relevant market research data is automatically displayed.

[0134] Example prompt:

[0135] Send reminder notifications at the start of the meeting, manage the scheduled time for each agenda item, and present relevant data in real time.

[0136] Recording of discussions and creation of meeting minutes

[0137] The terminal records the meeting content and sends the data to the server. The server uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the recording into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[0138] Example prompt:

[0139] Please record the meeting, transcribe the audio data into text, and automatically create and distribute meeting minutes.

[0140] Action plan generation and management

[0141] The server extracts action plans from meeting minutes and automatically assigns them to the appropriate personnel. Natural language processing technology is used to identify important tasks and assign them to the correct individuals. Each user manages and updates the progress of their assigned action plan within the system. The server periodically reports the progress to all participants.

[0142] Example prompt:

[0143] Extract action plans from the meeting minutes, assign them to the responsible parties, and have them report on the progress regularly.

[0144] As described above, the system of the present invention can streamline and automate the entire process from preparation to implementation and follow-up of online meetings. Through this system, the effort and time required for meeting management can be significantly reduced.

[0145] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0146] Step 1:

[0147] The user enters and submits an online meeting request. The user fills in details such as the date and time, participant list, and agenda in the form and presses the "Submit" button. The input data includes the meeting date and time, participant list, and agenda.

[0148] Specific actions:

[0149] The user opens a web form.

[0150] Fill in the details of the meeting in the input field.

[0151] Click the "Send" button.

[0152] Output: Meeting request data received by the server.

[0153] Step 2:

[0154] The server receives requests from users and stores them in the database. The server receives HTTP POST requests, parses their contents, and adds new records to the database.

[0155] Specific actions:

[0156] The server waits for HTTP requests.

[0157] User request received.

[0158] The request content is analyzed and saved to the "Meeting Requests" table.

[0159] Input: Meeting request data from the user.

[0160] Output: Meeting request information stored in the database.

[0161] Step 3:

[0162] The server communicates with all participants' calendar APIs to retrieve calendar information. The server uses each participant's authentication token to access the calendar API and retrieve event information.

[0163] Specific actions:

[0164] The server sends a request to the calendar API.

[0165] Retrieve the calendar information of each participant.

[0166] Analyze the calendar information to list available dates and times.

[0167] Input: Participant authentication token.

[0168] Output: Participant calendar information.

[0169] Step 4:

[0170] Based on the calendar information obtained by the server, the optimal date and time for all participants to attend is calculated. These suggested dates and times are then notified to the users.

[0171] Specific actions:

[0172] Analyze the acquired calendar information.

[0173] The algorithm calculates the dates and times when everyone is free.

[0174] The notification sending function will send the user a list of possible dates and times.

[0175] Input: Participant's calendar information.

[0176] Output: A list of possible dates and times sent to the user.

[0177] Step 5:

[0178] The user reviews the suggested dates and times and approves the most suitable one. Upon receiving the notification, the user selects the appropriate date and time from the list of options and replies to the system.

[0179] Specific actions:

[0180] The user opens the notification.

[0181] Check the suggested dates and times.

[0182] Select a date and time for approval and reply to the system.

[0183] Input: A list of possible dates and times from the server.

[0184] Output: Final date and time of approval.

[0185] Step 6:

[0186] The device collects meeting-related materials and data. It accesses shared folders (e.g., Google Drive, Dropbox) and external databases to retrieve and organize necessary materials.

[0187] Specific actions:

[0188] The device accesses shared folders and external databases.

[0189] Download the necessary documents.

[0190] Classify and organize the documents.

[0191] Input: Access information for data collection.

[0192] Output: Collected and organized materials.

[0193] Step 7:

[0194] The server uploads the collected materials to a shared folder, creates a meeting minutes template based on the agenda, and distributes it to the participants.

[0195] Specific actions:

[0196] The server uploads the documents to the shared folder.

[0197] Create a meeting minutes template using the template generation module.

[0198] Distribute the generated template to participants via email or shared link.

[0199] Input: Collected and organized materials.

[0200] Output: Documents uploaded to the shared folder, meeting minutes templates distributed to participants.

[0201] Step 8:

[0202] The server will send a reminder notification when the meeting starts.

[0203] Specific actions:

[0204] The server executes code that sends a reminder 5 minutes before the start time.

[0205] A reminder notification will be sent to each participant's device.

[0206] Input: Meeting start time information.

[0207] Output: Reminder notifications sent to participants.

[0208] Step 9:

[0209] The server manages the scheduled meeting time and keeps track of it. During the meeting, it displays relevant data to the user's terminal in real time, depending on the agenda.

[0210] Specific actions:

[0211] The server sets a timer and manages the scheduled time.

[0212] Timekeeping alerts are sent to each user's device.

[0213] Relevant data will be delivered in real time.

[0214] Input: Scheduled time information for each agenda item.

[0215] Output: Real-time data displayed on the user's device.

[0216] Step 10:

[0217] The device records the meeting content and sends the data to the server.

[0218] Specific actions:

[0219] The device launches the voice recording app.

[0220] Record the meeting content and save it to local storage.

[0221] Upload the recording file to the server.

[0222] Input: Audio data from a meeting.

[0223] Output: Recording file uploaded to the server.

[0224] Step 11:

[0225] The server uses a speech recognition API to convert the audio data into text and generate meeting minutes.

[0226] Specific actions:

[0227] The server sends audio data to the speech recognition API.

[0228] The acquired text data is formatted and compiled into meeting minutes.

[0229] Input: Audio file.

[0230] Output: Generated meeting minutes.

[0231] Step 12:

[0232] The server extracts action plans from meeting minutes and automatically assigns them to responsible personnel. The server uses natural language processing technology to identify important tasks and notifies the responsible personnel.

[0233] Specific actions:

[0234] The server analyzes the meeting minutes and extracts the tasks.

[0235] Assign the extracted tasks to the responsible personnel.

[0236] Input: Generated meeting minutes.

[0237] Output: Action plan assigned to the person in charge.

[0238] Step 13:

[0239] Users manage the progress of their assigned action plans and update them through the system as needed.

[0240] Specific actions:

[0241] The user logs into the system.

[0242] Use the interface to update the progress of the action plan.

[0243] Update the progress status.

[0244] Input: Progress status of the action plan.

[0245] Output: Updated progress.

[0246] Step 14:

[0247] The server will periodically report the progress of the action plan to all participants.

[0248] Specific actions:

[0249] The server checks the progress at regular intervals.

[0250] Summarize the progress and generate a report.

[0251] Send the report to all participants.

[0252] Input: Updated progress status.

[0253] Output: Progress report sent to all participants.

[0254] (Application Example 1)

[0255] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0256] Traditional online meeting systems faced challenges such as requiring significant time and effort for meeting preparation, execution, and follow-up. Furthermore, recording meetings and creating minutes were cumbersome, making efficient operation difficult. This was especially true when multiple participants were involved, as smoothly managing all processes—scheduling, organizing materials, automatically generating minutes, and managing action plans—became even more challenging. As a result, the quality of meetings often suffered, hindering the maximization of results.

[0257] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0258] In this invention, the server includes means for receiving meeting requests, means for obtaining calendar information of meeting participants and calculating the optimal date and time, means for collecting meeting-related materials and data, organizing them in a shared folder and uploading them, means for managing scheduled times for each agenda item and presenting relevant data to the user terminal in real time, means for recording meeting content, automatically transcribing it into text and generating meeting minutes, means for automatically extracting action plans from meeting minutes and assigning them to responsible persons, means for communicating with participants' calendars to calculate the optimal date and time and notifying the user, means for collecting and organizing relevant data and sharing a webinar page automatically created using a template, means for providing relevant information in real time during the meeting, and means for inputting prompt sentences using a generation AI model and transcribing audio data into text. This makes it possible to streamline and automate the entire process from meeting preparation to progress and follow-up.

[0259] "Means of receiving meeting requests" refers to a function where a user proposes an online meeting, sends detailed information such as the date and time, participant list, and agenda to the system, and the server receives the request.

[0260] The "method for obtaining meeting participants' calendar information and calculating the optimal date and time" is a function that communicates with each meeting participant's calendar API, checks everyone's availability, and calculates the most suitable date and time for the meeting.

[0261] "A means of collecting meeting-related materials and data, organizing them in a shared folder, and uploading them" refers to a function that collects necessary materials and data from different sources, organizes them, and uploads them to a single shared folder.

[0262] "A means of managing scheduled time for each agenda item and presenting relevant data to the user's terminal in real time" refers to a function that manages the scheduled time allocated to each agenda item and displays relevant data on the user's terminal in real time during the meeting.

[0263] "A method for recording meeting content, automatically converting it to text, and generating meeting minutes" refers to a function that records audio during a meeting, automatically converts that audio data into text, and generates meeting minutes.

[0264] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a function that extracts specific action plans from generated meeting minutes and automatically assigns them to the responsible personnel.

[0265] "A means of communicating with participants' calendars to calculate the optimal date and time and notifying the user" refers to a function that communicates with meeting participants' calendar information, calculates multiple date and time options, and notifies the user.

[0266] "A means of collecting and organizing relevant data and sharing a webinar page automatically created using a template" refers to a function that collects the necessary data related to a webinar, organizes it, and then automatically generates and shares a webinar page using a template.

[0267] "Means of providing relevant information in real time during a meeting" refers to a function that provides information and materials related to the agenda in real time while the meeting is in progress.

[0268] "A means of inputting prompt sentences using a generative AI model and converting audio data to text" refers to a function that utilizes a generative AI model to input prompt sentences for converting audio data to text, and then converts the audio data to text based on these prompts.

[0269] The present invention provides a multi-functional system that enables the efficient and automated operation of online meetings. In this system, the server, terminals, and users each play specific roles and work together.

[0270] System Configuration

[0271] hardware

[0272] Smartphone (iOS / ANDROID (registered trademark)): The device used by users and meeting participants

[0273] Server: The central computer that processes and manages data

[0274] Internet connection: Responsible for communication between all devices

[0275] Software

[0276] Client side: React Native

[0277] Server side: Node.js, Express

[0278] Database: MongoDB

[0279] AI voice recognition: Google Cloud Speech-to-Text API

[0280] Calendar API: Google Calendar API, Microsoft (registered trademark) Outlook API

[0281] Function flow

[0282] 1. Means of receiving meeting requests

[0283] The user uses the application on the smartphone to send a request for an online meeting to the server. This request includes detailed information such as the meeting date and time, participant list, and topics. The server receives this information and saves it in the database.

[0284] 2. Means of obtaining the calendar information of meeting participants and calculating the optimal date and time

[0285] The server communicates with the calendar APIs (Google Calendar API or Microsoft Outlook API) of each meeting participant to obtain the free time of all participants. Based on the obtained calendar information, the server calculates the optimal date and time and notifies the user.

[0286] 3. Means for collecting materials and data related to the meeting, organizing them in a shared folder, and uploading them

[0287] The user's terminal collects the materials and data required for the meeting, and the server organizes and uploads them to the shared folder. This shared folder is accessible to the user and meeting participants.

[0288] 4. Means for managing the scheduled time for each topic and presenting relevant data to the user terminal in real time

[0289] The server manages the scheduled time for each topic during the meeting and tracks the passage of time. It also presents relevant data to the user's smartphone in real time.

[0290] 5. Means for recording the meeting content, automatically converting it to text, and generating meeting minutes

[0291] The user's smartphone records the meeting content and transmits the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio data to text and automatically generates meeting minutes.

[0292] 6. Means for automatically extracting an action plan from the meeting minutes and assigning it to the responsible person

[0293] The server extracts the action plan from the generated meeting minutes and assigns it to the responsible person. The user manages the progress of the action plan based on this.

[0294] 7. Means for providing relevant information in real time

[0295] The server provides users with relevant information and materials in real time during the meeting, based on the agenda. This feature ensures that the meeting proceeds smoothly.

[0296] 8. A method for inputting prompt text using a generative AI model and converting speech data into text.

[0297] Users can leverage a generative AI model to input appropriate prompts and efficiently convert audio data into text.

[0298] Specific example

[0299] When a company holds an online webinar to announce a new product, this system automates and streamlines all preparation and execution. For example, a user submits a request for a new product announcement webinar to the system, and the server retrieves each participant's calendar information to calculate the optimal date and time. Relevant materials are collected, and the webinar page is automatically generated based on an organized template. During the meeting, relevant information is provided in real time, and after the meeting, the audio data is transcribed into text to generate meeting minutes.

[0300] Examples of prompts to input into a generative AI model:

[0301] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new product features, market strategy, and Q&A after the session. The audio recording is available below."

[0302] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0303] Step 1:

[0304] The user sends a request for an online meeting from the smartphone app. As input, detailed information such as the meeting date and time, participant list, and topics is entered. As output, that information is sent to the server and stored in the database. At this time, the format stored in the database includes fields such as meeting ID, date and time, participant list, and topics. Specifically, the user fills in the necessary information in the app's input form and presses the send button.

[0305] Step 2:

[0306] Based on the meeting information stored in the database, the server obtains the calendar information of the participants. As input, the server sends requests to the calendar APIs (such as Google Calendar API and Microsoft Outlook API) of each participant based on the meeting ID and participant list. As output, the free time of each participant is returned to the server. Specifically, the server obtains the calendar information of each participant through parallel processing and summarizes the free time of all participants in a list.

[0307] Step 3:

[0308] Based on the obtained calendar information, the server calculates the optimal meeting date and time. As input, it receives the list of free times of the participants' calendar information. As output, it generates a list of the optimal dates and times when all participants are free. Specifically, the server executes the logic of comparing the list of free times and selecting the date and time when the most participants are free.

[0309] Step 4:

[0310] The server notifies the user of the calculated optimal date and time. As input, it generates a notification message based on the list of optimal dates and times. As output, that notification message is sent to the user's smartphone and the user can view it. Specifically, the server sends a message to the smartphone through the push notification API.

[0311] Step 5:

[0312] The terminal collects the necessary materials and data for the meeting and sends them to the server. As input, it receives access links to materials provided by users and meeting participants, as well as uploaded files. As output, the materials and data are uploaded to the server and organized in a shared folder. Specifically, users use the app's material upload function to select and submit the necessary files.

[0313] Step 6:

[0314] The server organizes and uploads received documents and data to a shared folder. It receives collected documents and data as input. As output, these are categorized and saved in a designated folder, making them accessible to participants. Specifically, the server automatically generates a directory structure and appropriately places the documents.

[0315] Step 7:

[0316] The server manages the scheduled time and agenda during a meeting. It receives the start time and the scheduled time for each agenda item as input. Its output includes real-time progress tracking and displays warnings as the scheduled end time approaches. Specifically, the server uses a timer function to monitor time and issues alerts each time the scheduled time is reached.

[0317] Step 8:

[0318] The server records the meeting content and sends the audio data to the Google Cloud Speech-to-Text API for transcription. It receives the recorded audio data as input and automatically generated text data as output. Specifically, the server sends the audio file to the API and converts the returned text data into meeting minutes.

[0319] Step 9:

[0320] The server extracts action plans from meeting minutes and assigns them to responsible personnel. It analyzes the transcribed meeting minutes as input. The output determines each action item and its assigned person. Specifically, the server uses natural language processing to analyze the meeting minutes and automatically generates and assigns action plans.

[0321] Step 10:

[0322] The server uses a generative AI model to take prompt text as input and provides a means to convert audio data into text. As input, audio data is sent to the model along with the prompt text. As output, text data based on the prompt is generated. Specifically, the server appropriately constructs the prompt text and performs immediate analysis on the AI ​​model.

[0323] Examples of prompts to input into a generative AI model:

[0324] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new product features, market strategy, and Q&A after the session. The audio recording is available below."

[0325] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0326] The system of the present invention provides a combination of multiple functions for efficiently and highly automating online meetings. In particular, the quality of meetings can be further improved by incorporating an emotion engine. This system is centered around a server, terminals, and users, each playing a specific role.

[0327] 1. Accepting requests

[0328] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0329] The server receives the request and saves it to the database. This data is used for subsequent processing.

[0330] 2. Scheduling

[0331] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[0332] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[0333] These proposed dates and times will be notified to the users for each participant to review. The date and time will be finalized upon final approval.

[0334] 3. Meeting preparations

[0335] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0336] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[0337] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[0338] 4. Meeting Management

[0339] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[0340] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[0341] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[0342] 5. Utilizing the Emotion Engine

[0343] The emotion engine analyzes camera footage and audio data from meeting participants to recognize each participant's emotional state in real time.

[0344] The server adjusts the meeting's progress based on emotional data obtained from the emotion engine. For example, if a participant's concentration is waning, it can send a notification prompting them to take a break.

[0345] The server incorporates emotional data into the meeting minutes, recording the emotional state in relation to the topic or statement in question.

[0346] 6. Recording of discussions and creation of meeting minutes.

[0347] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[0348] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[0349] The server automatically generates meeting minutes based on the generated text. The minutes are organized according to a pre-distributed template. Sentimental data is also recorded, making it easier to understand the emotional context of the statements.

[0350] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[0351] 7. Action plan generation and management

[0352] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[0353] Users manage the progress of their assigned action plans. They update the task progress on the system.

[0354] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[0355] Specific example

[0356] For example, suppose a company wants to hold an online meeting for a new product development project. In this case, the user proposes the meeting regarding the new product development, enters the date and time, participant list, and agenda into the system, and submits the request.

[0357] The server receives the request and retrieves each participant's calendar information. It calculates the optimal date and time and notifies everyone. All participants review the proposed dates and times and finalize the date and time.

[0358] Before the meeting, terminals collect market research data and competitor analysis materials, and the server organizes and uploads them to a shared folder. Additionally, meeting minute templates tailored to the agenda are generated and distributed to participants in advance.

[0359] During the meeting, the server keeps track of the time and displays relevant data in real time for each agenda item. An emotion engine analyzes the emotional state of participants and sends timely notifications for breaks or adjustments to the meeting's progress. The meeting content is recorded by the terminals, and the server converts the audio data into text to automatically create meeting minutes. Emotional data is also recorded in the minutes.

[0360] After the meeting ends, the server extracts action plans from the meeting minutes and assigns them to the responsible parties. Users manage and update the progress of the action plans within the system, and the server reports the progress to all participants.

[0361] As a result, it becomes possible to streamline and automate the entire process from meeting preparation to execution and follow-up, and to improve the quality of meetings by utilizing sentiment data.

[0362] The following describes the processing flow.

[0363] Step 1:

[0364] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0365] Step 2:

[0366] The server receives this request and saves it to the database. This data is used for subsequent processing.

[0367] Step 3:

[0368] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[0369] Step 4:

[0370] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[0371] Step 5:

[0372] The server notifies users of a list of possible dates and times, and each participant confirms them. Based on the participants' confirmations, the optimal date and time are finalized.

[0373] Step 6:

[0374] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0375] Step 7:

[0376] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[0377] Step 8:

[0378] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[0379] Step 9:

[0380] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[0381] Step 10:

[0382] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[0383] Step 11:

[0384] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[0385] Step 12:

[0386] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[0387] Step 13:

[0388] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[0389] Step 14:

[0390] The emotion engine analyzes camera footage and audio data from meeting participants to recognize each participant's emotional state in real time.

[0391] Step 15:

[0392] The server adjusts the meeting's progress based on emotional data obtained from the emotion engine. For example, if a participant's concentration is waning, it can send a notification prompting them to take a break.

[0393] Step 16:

[0394] The server uses emotion data to reflect the emotional state of the participants in the meeting minutes.

[0395] Step 17:

[0396] The server creates meeting minutes based on automatically generated text. The minutes, organized according to a pre-distributed template, also reflect sentiment data.

[0397] Step 18:

[0398] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[0399] Step 19:

[0400] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[0401] Step 20:

[0402] Manage the progress of action plans assigned to users. Update task progress on the system.

[0403] Step 21:

[0404] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[0405] The above is the specific processing flow of the program for this system.

[0406] (Example 2)

[0407] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0408] In recent years, online meetings have become commonplace, but they require a great deal of manual intervention, and efficient meeting management and follow-up are essential. In particular, coordinating schedules for busy participants, sharing materials, creating meeting minutes, and managing participants' emotional states are all complex and time-consuming tasks. Managing post-meeting action plans and reporting progress also presents challenges. A system is needed to address these issues and streamline and enhance online meetings.

[0409] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving meeting requests, means for acquiring time management information of meeting participants and calculating the optimal date and time, means for collecting and sharing materials and data related to the meeting, means for managing the scheduled time for each agenda item during the meeting and presenting relevant data in real time, means for recording the meeting content, automatically converting it into text data to generate meeting minutes, means for automatically extracting action plans from the meeting minutes and assigning them to the responsible persons, means for recognizing the emotional state of meeting participants using a recording device and adjusting the progress of the meeting, and means for reflecting the emotional state of participants in response to their statements in the meeting minutes. This streamlines and automates the entire process from preparation to progress and follow-up of online meetings, enabling flexible meeting management that takes emotional states into consideration.

[0410] "Means for receiving meeting requests" refers to a function that receives requests from users that include detailed information such as the date and time of the meeting, participant list, and agenda, and stores this information in a database.

[0411] "Means for obtaining time management information of meeting participants" refers to a function that obtains the calendar information of meeting participants and allows for understanding each participant's schedule and available time.

[0412] The "means for calculating the optimal date and time" refers to a function that executes an algorithm based on acquired time management information to calculate the date and time when the largest number of participants are available.

[0413] "Means for collecting and sharing meeting-related materials and data" refers to the function of collecting materials and data necessary for the meeting agenda from external and internal databases and saving them in a shared folder in a format accessible to all participants.

[0414] "A means of managing scheduled time for each agenda item and presenting relevant data in real time" refers to a function that manages the scheduled time for each agenda item during a meeting and displays relevant data to participants in real time according to the progress.

[0415] "A method for recording meeting content and automatically converting it into text data to generate meeting minutes" refers to a function that records meeting content, automatically converts the recorded data into text data using speech recognition technology, and generates meeting minutes.

[0416] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a function that automatically extracts specific tasks from the content of the generated meeting minutes and assigns each task to the appropriate person.

[0417] "A means of recognizing the emotional state of meeting participants using a recording device and adjusting the progress of the meeting" refers to a function that analyzes the camera footage and audio of meeting participants, recognizes their emotional state in real time, and adjusts the progress of the meeting in a timely manner based on that data.

[0418] "Methods for reflecting emotional states in meeting minutes" refer to a function that records emotional data related to the content of statements in the minutes, thereby clarifying the emotional background of the statements.

[0419] The system of this invention provides a combination of multiple functions to efficiently and highly automate online meetings. The server, terminal, and user each play a specific role.

[0420] The server operates using the following software: MySQL® for database management, Apache® for the web server, and Tensorflow® for speech recognition and sentiment analysis. The terminal uses the Google Drive API and built-in microphone for collecting documents and recording meeting content. Users operate the system through its web interface, but use the Google Calendar API to retrieve calendar information.

[0421] Accepting requests

[0422] A user submits an online meeting request to the system. The request includes detailed information such as the date and time of the meeting, a list of participants, and the agenda. The server receives the request and stores it in a database (MySQL). This allows the request information to be centrally managed within the system and used for subsequent processing.

[0423] Schedule adjustment

[0424] The server communicates with the Google Calendar API to retrieve calendar information for all participants. Based on the retrieved calendar information, an algorithm calculates the optimal date and time. The server notifies each user of the calculated date and time candidates, and the final date and time are confirmed once the users review and approve the candidates. This allows for efficient scheduling among busy participants.

[0425] Meeting preparations

[0426] The terminal collects materials related to the meeting agenda via the Google Drive API. The server organizes the collected materials in a Docker container and uploads them to Google Drive. This ensures that the materials are shared in a format accessible to all participants. Furthermore, the server generates a meeting minutes template using a Python script and distributes it to participants.

[0427] Meeting progress

[0428] The server sends reminder notifications using Firebase Cloud Messaging one hour and ten minutes before the start of the meeting. During the meeting, a JavaScript® library is used for timekeeping, managing the scheduled time for each agenda item. In addition, relevant data is retrieved in real time from external APIs and internal databases and presented to participants. This streamlines the meeting process.

[0429] Using an Emotion Engine

[0430] The emotion engine analyzes participants' camera footage and audio, and uses TensorFlow to recognize their emotional state in real time. The server adjusts the meeting flow based on the emotion data and sends notifications prompting breaks if concentration levels decline. Furthermore, the emotion data is incorporated into the meeting minutes using a Python script.

[0431] Recording of discussions and creation of meeting minutes

[0432] The terminal records the meeting content and sends it to the server. The server uses a TensorFlow speech recognition model to transcribe the received recording into text and automatically creates meeting minutes. The minutes are distributed to all participants via email or in-system notifications. This ensures that the meeting content is recorded accurately and quickly.

[0433] Action plan generation and management

[0434] The server uses natural language processing technology to analyze meeting minutes and assign action plans to responsible personnel. Users manage and update progress on the system, and the server periodically reports the progress to all participants. This entire process streamlines post-meeting follow-up.

[0435] Examples of specific cases and prompt statements

[0436] When a company holds an online meeting for a new product development project, the user enters the date, time, participant list, and agenda into the system and sends a request. The server receives the request, retrieves each participant's calendar information, calculates the optimal date and time, and notifies them. All participants review the candidate dates and times and confirm the final date and time. Before the meeting, the terminal collects materials, the server organizes them, and uploads them to a shared folder. During the meeting, the server keeps track of the time and presents relevant data in real time. An emotion engine analyzes emotional states and prompts breaks at appropriate times. The meeting content is recorded by the terminal, and the server transcribes the audio data into text to create meeting minutes. After the meeting, the server extracts action plans from the minutes and assigns them to the responsible parties. The user manages the progress, and the server reports the progress to all participants.

[0437] Example of a prompt:

[0438] "Please explain how to use an online meeting system that utilizes an emotion engine. This system automates everything from scheduling and meeting management to minute-taking and action plan management."

[0439] As described above, the system of the present invention streamlines and automates the entire process from preparation to conducting and follow-up of online meetings, enabling flexible meeting management that takes into account emotional states.

[0440] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0441] Step 1:

[0442] A user submits an online meeting request to the system. The request includes detailed information such as the meeting date and time, participant list, and agenda. This information is processed as user input data and sent to the server. The server receives it as an HTTP POST request and stores it in a database (MySQL). The stored data is organized into fields such as meeting ID, meeting date and time, participant list, and agenda.

[0443] Step 2:

[0444] The server communicates with the Google Calendar API to retrieve calendar information for all participants. The server uses each user's API key to send requests to the Google Calendar API, retrieving each participant's schedule and availability. The retrieved calendar information is returned to the server in JSON format. The server parses this JSON data and lists each participant's availability.

[0445] Step 3:

[0446] The server calculates the optimal date and time based on the calendar information it has obtained. Using an algorithm, it identifies the date and time when the largest number of participants are available. Specifically, it calculates the overlap of each participant's available time slots and extracts the time slot that matches the most participants. This calculation result is generated as a list of candidate dates and times.

[0447] Step 4:

[0448] The server notifies each user of the calculated date and time options. The server sends the generated list of date and time options to each user using the email system or the system's notification function. Users review this and approve or modify it through the system interface or mobile app. The final confirmed date and time is sent to the server based on the user's input and stored in the database as a confirmed date and time list.

[0449] Step 5:

[0450] The device collects materials related to the meeting agenda. Specifically, it uses the Google Drive API to retrieve necessary materials from a shared folder. The device collects metadata for each material (file name, creation date, size, etc.) and sends it to the server. The server organizes the received metadata and generates a list of materials.

[0451] Step 6:

[0452] The server organizes the collected materials and uploads them to a shared folder. The server then analyzes the list of materials, categorizes them by importance, and re-uploads them to Google Drive. This ensures that the materials are shared in a format accessible to all participants. Simultaneously, the server generates a meeting minutes template and distributes it to all participants in advance. A Python script is used to automatically generate the template, including slots for each agenda item.

[0453] Step 7:

[0454] The server sends reminder notifications one hour and ten minutes before the meeting starts. Firebase Cloud Messaging is used to send these reminder notifications to each participant's device. This ensures that users are prepared and don't forget the meeting start time.

[0455] Step 8:

[0456] The server provides a timekeeping function during meetings. Using a JavaScript library, it manages the scheduled time for each agenda item in real time. The server monitors the progress of each agenda item and displays the remaining time on the screen. This ensures smooth meeting progress.

[0457] Step 9:

[0458] The server presents relevant data in real time. It retrieves necessary data from external APIs and internal databases and displays it to participants during the meeting. This ensures that discussions are data-driven and decision-making is accelerated.

[0459] Step 10:

[0460] The emotion engine runs a deep learning model using TensorFlow to analyze the camera footage and audio of meeting participants. It recognizes emotional states in real time and classifies them as positive, negative, neutral, etc. This data is then sent to a server.

[0461] Step 11:

[0462] The server adjusts the meeting flow based on emotional data. If participants' concentration levels decline, the server sends a notification prompting them to take a break. This improves the effectiveness of the meeting.

[0463] Step 12:

[0464] The server generates meeting minutes based on emotional states. A Python script is used to incorporate emotional data into the minutes. This clarifies the emotional context of statements, which is useful for subsequent analysis.

[0465] Step 13:

[0466] The terminal records the meeting content and sends the data to the server. The recorded data is saved as an audio file. The server uses a TensorFlow speech recognition model to convert the recorded data into text and generate character data.

[0467] Step 14:

[0468] The server automatically generates meeting minutes based on the generated text data. It organizes the text according to a pre-distributed template to produce complete meeting minutes. The minutes are distributed to all participants via email and system notifications.

[0469] Step 15:

[0470] The server automatically extracts action plans from meeting minutes and assigns them to responsible personnel. Using natural language processing technology, it generates specific tasks from the meeting minutes and automatically assigns them to the appropriate individuals. This streamlines post-meeting task management.

[0471] Step 16:

[0472] Users manage and update the progress of their action plans on the system. Users input and update progress through the system's web interface, and this data is sent to the server in real time. The server periodically reports the progress to all participants. This ensures that everyone is aware of the latest status.

[0473] As described above, this system streamlines the entire online meeting process and enables flexible meeting management that takes into account emotional states.

[0474] (Application Example 2)

[0475] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0476] In modern brick-and-mortar stores, efficient and effective customer service is essential, but it is difficult for staff to grasp the emotional state and interests of individual customers in real time. Furthermore, proper scheduling and reminder functions are necessary to efficiently handle multiple customers simultaneously, but traditional tools lack sufficient integration of these functions, leading to decreased staff efficiency.

[0477] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0478] In this invention, the server includes means for receiving meeting requests, means for acquiring meeting participant schedule information and calculating the optimal date and time, means for collecting and sharing meeting-related materials and data, means for managing scheduled time for each agenda item during the meeting and presenting relevant data in real time, means for recording meeting content, automatically transcribing it into text to generate meeting minutes, means for automatically extracting implementation plans from meeting minutes and assigning them to responsible personnel, means for recognizing participants' emotional states in real time using an emotion analysis engine and optimizing the progress of the meeting using that data, and means for managing customer service schedules and providing reminder notifications. This makes it possible to grasp customers' emotional states in real time in physical stores and respond quickly to the optimal needs, and also enables scheduling and reminder functions for efficiently handling multiple customers.

[0479] "Means for receiving meeting requests" refers to a device or software for acquiring online meeting requests and processing that data within the system.

[0480] "Means for obtaining meeting participants' schedule information and calculating the optimal date and time" refers to a device or software that has the function of checking the participants' schedules and automatically calculating the optimal date and time when everyone can attend.

[0481] "Means for collecting and sharing materials and data related to a meeting" refers to devices or software that collect materials and data used in a meeting and make them accessible to all participants.

[0482] "A means of managing scheduled time for each agenda item during a meeting and presenting relevant data in real time" refers to a system that manages time in accordance with the progress of the meeting and displays necessary information in real time.

[0483] "Methods for recording meeting content, automatically transcribing it into text, and generating meeting minutes" refers to a device or software that records audio during a meeting, automatically converts that audio data into text, and organizes and saves it as meeting minutes.

[0484] "A method for automatically extracting implementation plans from meeting minutes and assigning them to responsible personnel" refers to a system that analyzes specific action plans from generated meeting minutes and automatically assigns them to individual responsible personnel.

[0485] "A means of recognizing participants' emotional states in real time using an emotion analysis engine and optimizing the progress of a meeting using that data" refers to a device or software that analyzes participants' emotions in real time and adjusts and optimizes the progress of a meeting based on that emotional data.

[0486] "A means of managing customer service schedules and sending reminder notifications" refers to a system that manages customer service schedules and automatically sends reminder notifications.

[0487] The system of the present invention is designed to streamline and effectively handle customer interactions in physical stores, and is particularly characterized by its use of an emotion analysis engine to recognize the customer's emotional state in real time and optimize responses based on that recognition. Specific embodiments of this system will be described in detail below.

[0488] First, the server has a means of receiving online meeting requests. This means the meeting request is retrieved, and the data is processed within the system. This request includes detailed information such as the date and time, participant list, and agenda. The server stores this information in a database for use in subsequent processing.

[0489] Next, the server has a means to obtain the schedule information of meeting participants and calculate the optimal date and time. Using this means, the server communicates with each participant's calendar API to obtain their schedule information. Based on the obtained information, it calculates the date and time when the largest number of participants are available and notifies the user of the candidate date and time.

[0490] Furthermore, the server has a means of collecting and sharing materials and data related to the meeting. Through this means, the server collects necessary materials from shared folders and external databases, organizes them, and uploads them to the shared folder in a format accessible to all participants.

[0491] During the meeting, the server manages the scheduled time for each agenda item and has a means to display relevant data in real time. Using this means, the server manages the time in accordance with the progress of the meeting and displays the necessary information in real time. This allows participants to access the necessary data immediately.

[0492] Furthermore, the server has a means to record meeting content, automatically transcribe it into text, and generate meeting minutes. This means the server records the audio during the meeting and automatically transcribes the audio data into text. The generated text is then organized into meeting minutes according to a pre-distributed template. The server also has a means to automatically extract action plans from the meeting minutes and assign them to the responsible parties. This means the server analyzes the action plans from the generated minutes and automatically assigns them to the responsible parties.

[0493] The system also includes a means to use an emotion analysis engine to recognize participants' emotional states in real time and use that data to optimize the meeting's progress. This means the server can acquire emotional data in real time and appropriately adjust the meeting's progress, for example, by prompting participants to take a break if their concentration is waning.

[0494] Finally, the server has a means of managing customer service schedules and sending reminder notifications. This allows the server to manage customer service appointments and automatically send reminder notifications, enabling staff to efficiently handle multiple customers.

[0495] As a specific example, when a staff member wearing smart glasses efficiently attends to multiple customers, the following prompt message will be displayed.

[0496] "Current customers want to learn more about the products they've shown interest in. Provide them with relevant information and, if necessary, offer customized suggestions."

[0497] This system allows businesses to understand customers' emotional states in real time at physical stores, enabling them to respond quickly and optimally, thereby improving the quality of customer service.

[0498] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0499] Step 1:

[0500] Input: The user submits a request for an online meeting.

[0501] Action: The server receives this request and saves its contents to the database. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0502] Output: Request information is saved in the database.

[0503] Step 2:

[0504] Input: The server accesses each user's calendar API to retrieve the meeting participants' schedule information.

[0505] Operation: The server retrieves the schedule information of each participant and uses that information to calculate the date and time when the largest number of participants are available.

[0506] Output: The server notifies the user of the list of optimal date and time options.

[0507] Step 3:

[0508] Input: The server notifies the user of possible dates and times, and the user confirms the final date and time.

[0509] Action: The date and time confirmed by the user will be finalized as the meeting date and time.

[0510] Output: The final date and time are confirmed and notified to meeting participants.

[0511] Step 4:

[0512] Input: The server accesses shared folders and external databases to collect materials and data related to the meeting agenda.

[0513] Operation: The server organizes the collected data and uploads it to a shared folder in a format accessible to all participants.

[0514] Output: Necessary documents and data are saved in a shared folder.

[0515] Step 5:

[0516] Input: At the start of the meeting, the server sends a reminder notification to all participants.

[0517] Function: Sends reminder notifications to multiple devices to encourage meeting preparations.

[0518] Output: A reminder notification will be sent to all participants before the meeting starts.

[0519] Step 6:

[0520] Input: The server manages the scheduled time for each agenda item during the meeting and presents relevant data in real time.

[0521] Function: Manages time according to the progress of the meeting and displays necessary information in real time.

[0522] Output: Participants can immediately access the data they need.

[0523] Step 7:

[0524] Input: The server records the audio during the meeting.

[0525] Operation: The server uses speech recognition technology to automatically convert recorded audio data into text.

[0526] Output: The transcribed meeting content is generated as meeting minutes.

[0527] Step 8:

[0528] Input: The server automatically generates meeting minutes based on the generated text.

[0529] Operation: The server automatically generates meeting minutes and organizes them according to a pre-distributed template.

[0530] Output: Meeting minutes will be distributed to all participants via email and in-system notifications.

[0531] Step 9:

[0532] Input: The server uses an emotion analysis engine to recognize the emotional state of participants in real time.

[0533] Operation: The server uses sentiment data to optimize the meeting's progress and adjust the flow as needed. For example, it might prompt participants to take a break if their concentration is waning.

[0534] Output: Optimized meeting management is achieved.

[0535] Step 10:

[0536] Input: The server automatically extracts the implementation plan from the meeting minutes.

[0537] Operation: The server analyzes the implementation plan and assigns it to the responsible person.

[0538] Output: The implementation plan is assigned to the person in charge.

[0539] Step 11:

[0540] Input: The server retrieves data for managing customer service schedules.

[0541] Operation: The server manages customer service schedules and automatically sends reminder notifications.

[0542] Output: A reminder notification will be sent at the specified time.

[0543] The specific software used in each step includes MySQL for database management, the Google Calendar API for communication with the calendar API, the Google Cloud Speech-to-Text API for speech recognition, DeepFace for sentiment analysis, and Python program code for controlling the system.

[0544] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0545] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0546] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0547] [Second Embodiment]

[0548] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0549] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0550] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0551] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0552] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0553] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0554] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0555] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0556] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0557] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0558] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0559] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0560] The system of this invention combines multiple functions to enable the efficient and automated operation of online meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[0561] 1. Accepting requests

[0562] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0563] The server receives this request and saves it to the database. This data is used for subsequent processing.

[0564] 2. Scheduling

[0565] The server communicates with each user's calendar API to retrieve the calendar information of all participants.

[0566] Based on the acquired calendar information, the server calculates the optimal date and time. For example, it lists dates and times when many participants are available.

[0567] These proposed dates and times will be notified to the users for each participant to review. The date and time will be finalized upon final approval.

[0568] 3. Meeting preparations

[0569] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0570] The server organizes the collected data and uploads it to a shared folder.

[0571] Furthermore, the server creates meeting minutes templates based on the agenda items and distributes them to all participants.

[0572] 4. Meeting Management

[0573] The server sends a reminder notification to the user at the start of the meeting.

[0574] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data to the user's device in real time, depending on the agenda item.

[0575] 5. Recording of discussions and creation of meeting minutes.

[0576] The device records the meeting content and sends the data to the server.

[0577] The server converts the audio data into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[0578] 6. Action plan generation and management

[0579] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the responsible person.

[0580] Users manage the progress of their assigned action plans and update them through the system as needed.

[0581] The server will periodically report the progress of the action plan to all participants.

[0582] Specific example

[0583] For example, suppose a company wants to hold an online meeting for a new product development project. In this case, the user proposes the meeting regarding the new product development, enters the date and time, participant list, and agenda into the system, and submits the request.

[0584] The server receives the request and retrieves each participant's calendar information. It calculates the optimal date and time and notifies everyone. All participants review the proposed dates and times and finalize the date and time.

[0585] Before the meeting, terminals collect market research data and competitor analysis materials, and the server organizes and uploads them to a shared folder. Additionally, meeting minute templates tailored to the agenda are generated and distributed to participants in advance.

[0586] During the meeting, the server keeps track of the time and displays relevant data in real time for each agenda item. The meeting content is recorded by the terminals, and the server converts the audio data into text to automatically create meeting minutes.

[0587] After the meeting ends, the server extracts action plans from the meeting minutes and assigns them to the responsible parties. Users manage and update the progress of the action plans within the system, and the server reports the progress to all participants.

[0588] As a result, it becomes possible to streamline and automate the entire process, from meeting preparation to conducting and follow-up.

[0589] The following describes the processing flow.

[0590] Step 1:

[0591] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0592] Step 2:

[0593] The server receives this request and saves it to the database. This data is used for subsequent processing.

[0594] Step 3:

[0595] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[0596] Step 4:

[0597] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[0598] Step 5:

[0599] The server notifies users of a list of possible dates and times, and each participant confirms them. Based on the participants' confirmations, the optimal date and time are finalized.

[0600] Step 6:

[0601] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0602] Step 7:

[0603] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[0604] Step 8:

[0605] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[0606] Step 9:

[0607] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[0608] Step 10:

[0609] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[0610] Step 11:

[0611] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[0612] Step 12:

[0613] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[0614] Step 13:

[0615] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[0616] Step 14:

[0617] The server automatically generates meeting minutes based on the generated text. The minutes are organized according to a template distributed in advance.

[0618] Step 15:

[0619] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[0620] Step 16:

[0621] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[0622] Step 17:

[0623] Manage the progress of action plans assigned to users. Update task progress on the system.

[0624] Step 18:

[0625] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[0626] The above is the specific processing flow of the program for this system.

[0627] (Example 1)

[0628] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0629] Current online meeting systems make it difficult to efficiently manage the entire process, from meeting preparation to execution and follow-up. Furthermore, tasks such as timekeeping, real-time presentation of relevant materials during meetings, and post-meeting minute-taking and action plan extraction and management are not automated, relying heavily on manual processes. This results in significant effort and time being required for effective meeting management.

[0630] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0631] In this invention, the server includes means for receiving meeting requests, means for obtaining the schedule information of meeting participants and calculating the optimal date and time, means for collecting and sharing materials and information related to the meeting, means for managing the scheduled time for each agenda item during the meeting and presenting relevant information in real time, means for recording the meeting content, automatically transcribing it into text to generate meeting minutes, means for automatically extracting action plans from the meeting minutes and assigning them to responsible persons, means for sending reminders at the start of the meeting, means for sending meeting recording data to the server, and means for periodically reporting the progress of the action plans. This makes it possible to streamline and automate the entire process from meeting preparation to execution and follow-up.

[0632] "Means for receiving meeting requests" refers to the means by which users propose online meetings and send detailed information such as the date and time, participant list, and agenda to the server.

[0633] "A means of obtaining the schedule information of meeting participants and calculating the optimal date and time" refers to a method in which the server communicates with each participant's calendar API and calculates the optimal date and time that all participants can attend based on the information obtained.

[0634] "Means for collecting and sharing materials and information related to a meeting" refers to a method by which terminals collect necessary materials for a meeting from shared folders or external databases, and a server organizes and shares them with participants.

[0635] "A means of managing scheduled time for each agenda item during a meeting and presenting relevant information in real time" refers to a method in which a server manages the scheduled time during the meeting and presents relevant materials and information in real time as needed.

[0636] "A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes" refers to a method in which a terminal records the audio during a meeting, and a server uses speech recognition technology to transcribe that data into text and compile it into meeting minutes.

[0637] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a method in which a server analyzes the text of the meeting minutes, extracts specific action items, and automatically assigns them to the appropriate personnel.

[0638] "Method for sending reminders at the start of a meeting" refers to a method by which the server sends a reminder to all participants to notify them of the start time as the meeting is about to begin.

[0639] "Means for sending meeting recording data to the server" refers to the means by which a terminal uploads the recorded meeting audio data to the server.

[0640] "Means for periodically reporting on the progress of the action plan" refers to a method by which the server periodically checks the implementation status of the extracted action plan and reports its progress to all participants.

[0641] The system of the present invention enables the efficient and automated operation of online meetings through the collaborative roles of the server, terminal, and user. Specific embodiments are described below.

[0642] Accepting requests

[0643] A user enters and submits an online meeting request. This includes details such as the date and time, participant list, and agenda. For example, suppose a user submits a request for "Meeting on October 15, 2023 at 2:00 PM, with participants A, B, and C, on the topic of new product development." The server receives this request and stores it in the database. This makes the meeting details available for subsequent processing.

[0644] Example prompt:

[0645] I would like to hold a meeting regarding new product development on October 15, 2023, at 2:00 PM. The participants are Person A, Person B, and Person C. The topic of discussion will be "New Product Development".

[0646] Schedule adjustment

[0647] The server accesses the calendar APIs of all participants (e.g., Google Calendar API, Outlook Calendar API) to retrieve calendar information. Based on this information, the server calculates the optimal date and time and notifies the users of the suggested dates and times. Each user who receives the notification reviews the suggested dates and times and accepts the optimal one. Through this confirmation, the final date and time are determined.

[0648] Example prompt:

[0649] Please obtain each participant's calendar information, calculate a date and time that works for everyone, and notify them accordingly.

[0650] Meeting preparations

[0651] The terminal collects materials and data related to the meeting. These are collected from shared folders (e.g., Google Drive, Dropbox) or external databases (e.g., a company's internal database). The server organizes these materials and uploads them to the shared folder, as well as generating meeting minutes templates based on the agenda and distributing them to participants in advance.

[0652] Example prompt:

[0653] Collect market research data and competitive analysis materials, upload them to a shared folder, and generate a meeting minutes template to distribute to participants.

[0654] Meeting progress

[0655] The server sends a reminder notification at the start of the meeting. During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data related to the agenda item on the user's device in real time. For example, during a discussion on new product development, relevant market research data is automatically displayed.

[0656] Example prompt:

[0657] Send reminder notifications at the start of the meeting, manage the scheduled time for each agenda item, and present relevant data in real time.

[0658] Recording of discussions and creation of meeting minutes

[0659] The terminal records the meeting content and sends the data to the server. The server uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the recording into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[0660] Example prompt:

[0661] Please record the meeting, transcribe the audio data into text, and automatically create and distribute meeting minutes.

[0662] Action plan generation and management

[0663] The server extracts action plans from meeting minutes and automatically assigns them to the appropriate personnel. Natural language processing technology is used to identify important tasks and assign them to the correct individuals. Each user manages and updates the progress of their assigned action plan within the system. The server periodically reports the progress to all participants.

[0664] Example prompt:

[0665] Extract action plans from the meeting minutes, assign them to the responsible parties, and have them report on the progress regularly.

[0666] As described above, the system of the present invention can streamline and automate the entire process from preparation to implementation and follow-up of online meetings. Through this system, the effort and time required for meeting management can be significantly reduced.

[0667] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0668] Step 1:

[0669] The user enters and submits an online meeting request. The user fills in details such as the date and time, participant list, and agenda in the form and presses the "Submit" button. The input data includes the meeting date and time, participant list, and agenda.

[0670] Specific actions:

[0671] The user opens a web form.

[0672] Fill in the details of the meeting in the input field.

[0673] Click the "Send" button.

[0674] Output: Meeting request data received by the server.

[0675] Step 2:

[0676] The server receives requests from users and stores them in the database. The server receives HTTP POST requests, parses their contents, and adds new records to the database.

[0677] Specific actions:

[0678] The server waits for HTTP requests.

[0679] User request received.

[0680] The request content is analyzed and saved to the "Meeting Requests" table.

[0681] Input: Meeting request data from the user.

[0682] Output: Meeting request information stored in the database.

[0683] Step 3:

[0684] The server communicates with all participants' calendar APIs to retrieve calendar information. The server uses each participant's authentication token to access the calendar API and retrieve event information.

[0685] Specific actions:

[0686] The server sends a request to the calendar API.

[0687] Retrieve the calendar information of each participant.

[0688] Analyze the calendar information to list available dates and times.

[0689] Input: Participant authentication token.

[0690] Output: Participant calendar information.

[0691] Step 4:

[0692] Based on the calendar information obtained by the server, the optimal date and time for all participants to attend is calculated. These suggested dates and times are then notified to the users.

[0693] Specific actions:

[0694] Analyze the acquired calendar information.

[0695] The algorithm calculates the dates and times when everyone is free.

[0696] The notification sending function will send the user a list of possible dates and times.

[0697] Input: Participant's calendar information.

[0698] Output: A list of possible dates and times sent to the user.

[0699] Step 5:

[0700] The user reviews the suggested dates and times and approves the most suitable one. Upon receiving the notification, the user selects the appropriate date and time from the list of options and replies to the system.

[0701] Specific actions:

[0702] The user opens the notification.

[0703] Check the suggested dates and times.

[0704] Select a date and time for approval and reply to the system.

[0705] Input: A list of possible dates and times from the server.

[0706] Output: Final date and time of approval.

[0707] Step 6:

[0708] The device collects meeting-related materials and data. It accesses shared folders (e.g., Google Drive, Dropbox) and external databases to retrieve and organize necessary materials.

[0709] Specific actions:

[0710] The device accesses shared folders and external databases.

[0711] Download the necessary documents.

[0712] Classify and organize the documents.

[0713] Input: Access information for data collection.

[0714] Output: Collected and organized materials.

[0715] Step 7:

[0716] The server uploads the collected materials to a shared folder, creates a meeting minutes template based on the agenda, and distributes it to the participants.

[0717] Specific actions:

[0718] The server uploads the documents to the shared folder.

[0719] Create a meeting minutes template using the template generation module.

[0720] Distribute the generated template to participants via email or shared link.

[0721] Input: Collected and organized materials.

[0722] Output: Documents uploaded to the shared folder, meeting minutes templates distributed to participants.

[0723] Step 8:

[0724] The server will send a reminder notification when the meeting starts.

[0725] Specific actions:

[0726] The server executes code that sends a reminder 5 minutes before the start time.

[0727] A reminder notification will be sent to each participant's device.

[0728] Input: Meeting start time information.

[0729] Output: Reminder notifications sent to participants.

[0730] Step 9:

[0731] The server manages the scheduled meeting time and keeps track of it. During the meeting, it displays relevant data to the user's terminal in real time, depending on the agenda.

[0732] Specific actions:

[0733] The server sets a timer and manages the scheduled time.

[0734] Timekeeping alerts are sent to each user's device.

[0735] Relevant data will be delivered in real time.

[0736] Input: Scheduled time information for each agenda item.

[0737] Output: Real-time data displayed on the user's device.

[0738] Step 10:

[0739] The device records the meeting content and sends the data to the server.

[0740] Specific actions:

[0741] The device launches the voice recording app.

[0742] Record the meeting content and save it to local storage.

[0743] Upload the recording file to the server.

[0744] Input: Audio data from a meeting.

[0745] Output: Recording file uploaded to the server.

[0746] Step 11:

[0747] The server uses a speech recognition API to convert the audio data into text and generate meeting minutes.

[0748] Specific actions:

[0749] The server sends audio data to the speech recognition API.

[0750] The acquired text data is formatted and compiled into meeting minutes.

[0751] Input: Audio file.

[0752] Output: Generated meeting minutes.

[0753] Step 12:

[0754] The server extracts action plans from meeting minutes and automatically assigns them to responsible personnel. The server uses natural language processing technology to identify important tasks and notifies the responsible personnel.

[0755] Specific actions:

[0756] The server analyzes the meeting minutes and extracts the tasks.

[0757] Assign the extracted tasks to the responsible personnel.

[0758] Input: Generated meeting minutes.

[0759] Output: Action plan assigned to the person in charge.

[0760] Step 13:

[0761] Users manage the progress of their assigned action plans and update them through the system as needed.

[0762] Specific actions:

[0763] The user logs into the system.

[0764] Use the interface to update the progress of the action plan.

[0765] Update the progress status.

[0766] Input: Progress status of the action plan.

[0767] Output: Updated progress.

[0768] Step 14:

[0769] The server will periodically report the progress of the action plan to all participants.

[0770] Specific actions:

[0771] The server checks the progress at regular intervals.

[0772] Summarize the progress and generate a report.

[0773] Send the report to all participants.

[0774] Input: Updated progress status.

[0775] Output: Progress report sent to all participants.

[0776] (Application Example 1)

[0777] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0778] Traditional online meeting systems faced challenges such as requiring significant time and effort for meeting preparation, execution, and follow-up. Furthermore, recording meetings and creating minutes were cumbersome, making efficient operation difficult. This was especially true when multiple participants were involved, as smoothly managing all processes—scheduling, organizing materials, automatically generating minutes, and managing action plans—became even more challenging. As a result, the quality of meetings often suffered, hindering the maximization of results.

[0779] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0780] In this invention, the server includes means for receiving meeting requests, means for obtaining calendar information of meeting participants and calculating the optimal date and time, means for collecting meeting-related materials and data, organizing them in a shared folder and uploading them, means for managing scheduled times for each agenda item and presenting relevant data to the user terminal in real time, means for recording meeting content, automatically transcribing it into text and generating meeting minutes, means for automatically extracting action plans from meeting minutes and assigning them to responsible persons, means for communicating with participants' calendars to calculate the optimal date and time and notifying the user, means for collecting and organizing relevant data and sharing a webinar page automatically created using a template, means for providing relevant information in real time during the meeting, and means for inputting prompt sentences using a generation AI model and transcribing audio data into text. This makes it possible to streamline and automate the entire process from meeting preparation to progress and follow-up.

[0781] "Means of receiving meeting requests" refers to a function where a user proposes an online meeting, sends detailed information such as the date and time, participant list, and agenda to the system, and the server receives the request.

[0782] The "method for obtaining meeting participants' calendar information and calculating the optimal date and time" is a function that communicates with each meeting participant's calendar API, checks everyone's availability, and calculates the most suitable date and time for the meeting.

[0783] "A means of collecting meeting-related materials and data, organizing them in a shared folder, and uploading them" refers to a function that collects necessary materials and data from different sources, organizes them, and uploads them to a single shared folder.

[0784] "A means of managing scheduled time for each agenda item and presenting relevant data to the user's terminal in real time" refers to a function that manages the scheduled time allocated to each agenda item and displays relevant data on the user's terminal in real time during the meeting.

[0785] "A method for recording meeting content, automatically converting it to text, and generating meeting minutes" refers to a function that records audio during a meeting, automatically converts that audio data into text, and generates meeting minutes.

[0786] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a function that extracts specific action plans from generated meeting minutes and automatically assigns them to the responsible personnel.

[0787] "A means of communicating with participants' calendars to calculate the optimal date and time and notifying the user" refers to a function that communicates with meeting participants' calendar information, calculates multiple date and time options, and notifies the user.

[0788] "A means of collecting and organizing relevant data and sharing a webinar page automatically created using a template" refers to a function that collects the necessary data related to a webinar, organizes it, and then automatically generates and shares a webinar page using a template.

[0789] "Means of providing relevant information in real time during a meeting" refers to a function that provides information and materials related to the agenda in real time while the meeting is in progress.

[0790] "A means of inputting prompt sentences using a generative AI model and converting audio data to text" refers to a function that utilizes a generative AI model to input prompt sentences for converting audio data to text, and then converts the audio data to text based on these prompts.

[0791] The present invention provides a multi-functional system that enables the efficient and automated operation of online meetings. In this system, the server, terminals, and users each play specific roles and work together.

[0792] System Configuration

[0793] hardware

[0794] Smartphones (iOS / Android): Devices used by users and meeting participants.

[0795] Server: A central computer that processes and manages data.

[0796] Internet connection: Responsible for communication between all devices.

[0797] software

[0798] Client-side: React Native

[0799] Server-side: Node.js, Express

[0800] Database: MongoDB

[0801] AI speech recognition: Google Cloud Speech-to-Text API

[0802] Calendar APIs: Google Calendar API, Microsoft Outlook API

[0803] Function flow

[0804] 1. Means of receiving meeting requests

[0805] A user sends an online meeting request to the server using a smartphone application. This request includes detailed information such as the date and time of the meeting, a list of participants, and the agenda. The server receives this information and stores it in its database.

[0806] 2. A means of obtaining the calendar information of meeting participants and calculating the optimal date and time.

[0807] The server communicates with each meeting participant's calendar API (such as Google Calendar API or Microsoft Outlook API) to retrieve everyone's availability. Based on the retrieved calendar information, the server calculates the optimal date and time and notifies the user.

[0808] 3. A method for collecting meeting-related materials and data, organizing them in a shared folder, and uploading them.

[0809] The user's device collects the necessary materials and data for the meeting, and the server organizes them and uploads them to a shared folder. This shared folder is accessible to the user and meeting participants.

[0810] 4. A means of managing the scheduled time for each agenda item and presenting relevant data to the user's terminal in real time.

[0811] The server manages the scheduled time for each agenda item during the meeting and tracks the progress of time. It also displays relevant data to the user's smartphone in real time.

[0812] 5. A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes.

[0813] The user's smartphone records the meeting and sends the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio data into text and automatically generates meeting minutes.

[0814] 6. A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel.

[0815] The server extracts action plans from the generated meeting minutes and assigns them to responsible parties. Users then manage the progress of these action plans based on these plans.

[0816] 7. Means of providing relevant information in real time

[0817] The server provides users with relevant information and materials in real time during the meeting, based on the agenda. This feature ensures that the meeting proceeds smoothly.

[0818] 8. A method for inputting prompt text using a generative AI model and converting speech data into text.

[0819] Users can leverage a generative AI model to input appropriate prompts and efficiently convert audio data into text.

[0820] Specific example

[0821] When a company holds an online webinar to announce a new product, this system automates and streamlines all preparation and execution. For example, a user submits a request for a new product announcement webinar to the system, and the server retrieves each participant's calendar information to calculate the optimal date and time. Relevant materials are collected, and the webinar page is automatically generated based on an organized template. During the meeting, relevant information is provided in real time, and after the meeting, the audio data is transcribed into text to generate meeting minutes.

[0822] Examples of prompts to input into a generative AI model:

[0823] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new product features, market strategy, and Q&A after the session. The audio recording is available below."

[0824] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0825] Step 1:

[0826] A user sends an online meeting request via a smartphone app. The user enters detailed information such as the meeting date and time, participant list, and agenda. This information is then sent to a server and stored in a database. The database contains fields for meeting ID, date and time, participant list, and agenda. Specifically, the user fills in the required information in the app's input form and presses the submit button.

[0827] Step 2:

[0828] The server retrieves participants' calendar information based on meeting information stored in the database. As input, the server sends requests to each participant's calendar API (such as Google Calendar API or Microsoft Outlook API) based on the meeting ID and participant list. As output, the server receives each participant's free time. Specifically, the server retrieves each participant's calendar information in parallel and compiles everyone's free time into a list.

[0829] Step 3:

[0830] The server calculates the optimal meeting date and time based on the retrieved calendar information. It receives a list of participants' available time slots as input. As output, it generates a list of the optimal dates and times when all participants are available. Specifically, the server compares the available time slot lists and executes logic to select the date and time when the most participants are available.

[0831] Step 4:

[0832] The server notifies the user of the calculated optimal date and time. As input, it generates a notification message based on a list of optimal dates and times. As output, this notification message is sent to the user's smartphone, making it available for the user to view. Specifically, the server sends the message to the smartphone via a push notification API.

[0833] Step 5:

[0834] The terminal collects the necessary materials and data for the meeting and sends them to the server. As input, it receives access links to materials provided by users and meeting participants, as well as uploaded files. As output, the materials and data are uploaded to the server and organized in a shared folder. Specifically, users use the app's material upload function to select and submit the necessary files.

[0835] Step 6:

[0836] The server organizes and uploads received documents and data to a shared folder. It receives collected documents and data as input. As output, these are categorized and saved in a designated folder, making them accessible to participants. Specifically, the server automatically generates a directory structure and appropriately places the documents.

[0837] Step 7:

[0838] The server manages the scheduled time and agenda during a meeting. It receives the start time and the scheduled time for each agenda item as input. Its output includes real-time progress tracking and displays warnings as the scheduled end time approaches. Specifically, the server uses a timer function to monitor time and issues alerts each time the scheduled time is reached.

[0839] Step 8:

[0840] The server records the meeting content and sends the audio data to the Google Cloud Speech-to-Text API for transcription. It receives the recorded audio data as input and automatically generated text data as output. Specifically, the server sends the audio file to the API and converts the returned text data into meeting minutes.

[0841] Step 9:

[0842] The server extracts action plans from meeting minutes and assigns them to responsible personnel. It analyzes the transcribed meeting minutes as input. The output determines each action item and its assigned person. Specifically, the server uses natural language processing to analyze the meeting minutes and automatically generates and assigns action plans.

[0843] Step 10:

[0844] The server uses a generative AI model to take prompt text as input and provides a means to convert audio data into text. As input, audio data is sent to the model along with the prompt text. As output, text data based on the prompt is generated. Specifically, the server appropriately constructs the prompt text and performs immediate analysis on the AI ​​model.

[0845] Examples of prompts to input into a generative AI model:

[0846] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new product features, market strategy, and Q&A after the session. The audio recording is available below."

[0847] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0848] The system of the present invention provides a combination of multiple functions for efficiently and highly automating online meetings. In particular, the quality of meetings can be further improved by incorporating an emotion engine. This system is centered around a server, terminals, and users, each playing a specific role.

[0849] 1. Accepting requests

[0850] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0851] The server receives the request and saves it to the database. This data is used for subsequent processing.

[0852] 2. Scheduling

[0853] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[0854] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[0855] These proposed dates and times will be notified to the users for each participant to review. The date and time will be finalized upon final approval.

[0856] 3. Meeting preparations

[0857] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0858] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[0859] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[0860] 4. Meeting Management

[0861] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[0862] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[0863] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[0864] 5. Utilizing the Emotion Engine

[0865] The emotion engine analyzes camera footage and audio data from meeting participants to recognize each participant's emotional state in real time.

[0866] The server adjusts the meeting's progress based on emotional data obtained from the emotion engine. For example, if a participant's concentration is waning, it can send a notification prompting them to take a break.

[0867] The server incorporates emotional data into the meeting minutes, recording the emotional state in relation to the topic or statement in question.

[0868] 6. Recording of discussions and creation of meeting minutes.

[0869] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[0870] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[0871] The server automatically generates meeting minutes based on the generated text. The minutes are organized according to a pre-distributed template. Sentimental data is also recorded, making it easier to understand the emotional context of the statements.

[0872] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[0873] 7. Action plan generation and management

[0874] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[0875] Users manage the progress of their assigned action plans. They update the task progress on the system.

[0876] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[0877] Specific example

[0878] For example, suppose a company wants to hold an online meeting for a new product development project. In this case, the user proposes the meeting regarding the new product development, enters the date and time, participant list, and agenda into the system, and submits the request.

[0879] The server receives the request and retrieves each participant's calendar information. It calculates the optimal date and time and notifies everyone. All participants review the proposed dates and times and finalize the date and time.

[0880] Before the meeting, terminals collect market research data and competitor analysis materials, and the server organizes and uploads them to a shared folder. Additionally, meeting minute templates tailored to the agenda are generated and distributed to participants in advance.

[0881] During the meeting, the server keeps track of the time and displays relevant data in real time for each agenda item. An emotion engine analyzes the emotional state of participants and sends timely notifications for breaks or adjustments to the meeting's progress. The meeting content is recorded by the terminals, and the server converts the audio data into text to automatically create meeting minutes. Emotional data is also recorded in the minutes.

[0882] After the meeting ends, the server extracts action plans from the meeting minutes and assigns them to the responsible parties. Users manage and update the progress of the action plans within the system, and the server reports the progress to all participants.

[0883] As a result, it becomes possible to streamline and automate the entire process from meeting preparation to execution and follow-up, and to improve the quality of meetings by utilizing sentiment data.

[0884] The following describes the processing flow.

[0885] Step 1:

[0886] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[0887] Step 2:

[0888] The server receives this request and saves it to the database. This data is used for subsequent processing.

[0889] Step 3:

[0890] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[0891] Step 4:

[0892] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[0893] Step 5:

[0894] The server notifies users of a list of possible dates and times, and each participant confirms them. Based on the participants' confirmations, the optimal date and time are finalized.

[0895] Step 6:

[0896] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[0897] Step 7:

[0898] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[0899] Step 8:

[0900] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[0901] Step 9:

[0902] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[0903] Step 10:

[0904] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[0905] Step 11:

[0906] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[0907] Step 12:

[0908] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[0909] Step 13:

[0910] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[0911] Step 14:

[0912] The emotion engine analyzes camera footage and audio data from meeting participants to recognize each participant's emotional state in real time.

[0913] Step 15:

[0914] The server adjusts the meeting's progress based on emotional data obtained from the emotion engine. For example, if a participant's concentration is waning, it can send a notification prompting them to take a break.

[0915] Step 16:

[0916] The server uses emotion data to reflect the emotional state of the participants in the meeting minutes.

[0917] Step 17:

[0918] The server creates meeting minutes based on automatically generated text. The minutes, organized according to a pre-distributed template, also reflect sentiment data.

[0919] Step 18:

[0920] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[0921] Step 19:

[0922] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[0923] Step 20:

[0924] Manage the progress of action plans assigned to users. Update task progress on the system.

[0925] Step 21:

[0926] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[0927] The above is the specific processing flow of the program for this system.

[0928] (Example 2)

[0929] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0930] In recent years, online meetings have become commonplace, but they require a great deal of manual intervention, and efficient meeting management and follow-up are essential. In particular, coordinating schedules for busy participants, sharing materials, creating meeting minutes, and managing participants' emotional states are all complex and time-consuming tasks. Managing post-meeting action plans and reporting progress also presents challenges. A system is needed to address these issues and streamline and enhance online meetings.

[0931] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving meeting requests, means for acquiring time management information of meeting participants and calculating the optimal date and time, means for collecting and sharing materials and data related to the meeting, means for managing the scheduled time for each agenda item during the meeting and presenting relevant data in real time, means for recording the meeting content, automatically converting it into text data to generate meeting minutes, means for automatically extracting action plans from the meeting minutes and assigning them to the responsible persons, means for recognizing the emotional state of meeting participants using a recording device and adjusting the progress of the meeting, and means for reflecting the emotional state of participants in response to their statements in the meeting minutes. This streamlines and automates the entire process from preparation to progress and follow-up of online meetings, enabling flexible meeting management that takes emotional states into consideration.

[0932] "Means for receiving meeting requests" refers to a function that receives requests from users that include detailed information such as the date and time of the meeting, participant list, and agenda, and stores this information in a database.

[0933] "Means for obtaining time management information of meeting participants" refers to a function that obtains the calendar information of meeting participants and allows for understanding each participant's schedule and available time.

[0934] The "means for calculating the optimal date and time" refers to a function that executes an algorithm based on acquired time management information to calculate the date and time when the largest number of participants are available.

[0935] "Means for collecting and sharing meeting-related materials and data" refers to the function of collecting materials and data necessary for the meeting agenda from external and internal databases and saving them in a shared folder in a format accessible to all participants.

[0936] "A means of managing scheduled time for each agenda item and presenting relevant data in real time" refers to a function that manages the scheduled time for each agenda item during a meeting and displays relevant data to participants in real time according to the progress.

[0937] "A method for recording meeting content and automatically converting it into text data to generate meeting minutes" refers to a function that records meeting content, automatically converts the recorded data into text data using speech recognition technology, and generates meeting minutes.

[0938] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a function that automatically extracts specific tasks from the content of the generated meeting minutes and assigns each task to the appropriate person.

[0939] "A means of recognizing the emotional state of meeting participants using a recording device and adjusting the progress of the meeting" refers to a function that analyzes the camera footage and audio of meeting participants, recognizes their emotional state in real time, and adjusts the progress of the meeting in a timely manner based on that data.

[0940] "Methods for reflecting emotional states in meeting minutes" refer to a function that records emotional data related to the content of statements in the minutes, thereby clarifying the emotional background of the statements.

[0941] The system of this invention provides a combination of multiple functions to efficiently and highly automate online meetings. The server, terminal, and user each play a specific role.

[0942] The server operates using the following software: MySQL for database management, Apache as the web server, and TensorFlow for speech recognition and sentiment analysis. The terminal uses the Google Drive API and built-in microphone for collecting documents and recording meeting content. Users operate the system through its web interface, but use the Google Calendar API to retrieve calendar information.

[0943] Accepting requests

[0944] A user submits an online meeting request to the system. The request includes detailed information such as the date and time of the meeting, a list of participants, and the agenda. The server receives the request and stores it in a database (MySQL). This allows the request information to be centrally managed within the system and used for subsequent processing.

[0945] Schedule adjustment

[0946] The server communicates with the Google Calendar API to retrieve calendar information for all participants. Based on the retrieved calendar information, an algorithm calculates the optimal date and time. The server notifies each user of the calculated date and time candidates, and the final date and time are confirmed once the users review and approve the candidates. This allows for efficient scheduling among busy participants.

[0947] Meeting preparations

[0948] The terminal collects materials related to the meeting agenda via the Google Drive API. The server organizes the collected materials in a Docker container and uploads them to Google Drive. This ensures that the materials are shared in a format accessible to all participants. Furthermore, the server generates a meeting minutes template using a Python script and distributes it to participants.

[0949] Meeting progress

[0950] The server sends reminder notifications using Firebase Cloud Messaging one hour and ten minutes before the start of the meeting. During the meeting, a JavaScript library is used for timekeeping, managing the scheduled time for each agenda item. In addition, relevant data is retrieved in real time from external APIs and internal databases and presented to participants. This streamlines the meeting process.

[0951] Using an Emotion Engine

[0952] The emotion engine analyzes participants' camera footage and audio, and uses TensorFlow to recognize their emotional state in real time. The server adjusts the meeting flow based on the emotion data and sends notifications prompting breaks if concentration levels decline. Furthermore, the emotion data is incorporated into the meeting minutes using a Python script.

[0953] Recording of discussions and creation of meeting minutes

[0954] The terminal records the meeting content and sends it to the server. The server uses a TensorFlow speech recognition model to transcribe the received recording into text and automatically creates meeting minutes. The minutes are distributed to all participants via email or in-system notifications. This ensures that the meeting content is recorded accurately and quickly.

[0955] Action plan generation and management

[0956] The server uses natural language processing technology to analyze meeting minutes and assign action plans to responsible personnel. Users manage and update progress on the system, and the server periodically reports the progress to all participants. This entire process streamlines post-meeting follow-up.

[0957] Examples of specific cases and prompt statements

[0958] When a company holds an online meeting for a new product development project, the user enters the date, time, participant list, and agenda into the system and sends a request. The server receives the request, retrieves each participant's calendar information, calculates the optimal date and time, and notifies them. All participants review the candidate dates and times and confirm the final date and time. Before the meeting, the terminal collects materials, the server organizes them, and uploads them to a shared folder. During the meeting, the server keeps track of the time and presents relevant data in real time. An emotion engine analyzes emotional states and prompts breaks at appropriate times. The meeting content is recorded by the terminal, and the server transcribes the audio data into text to create meeting minutes. After the meeting, the server extracts action plans from the minutes and assigns them to the responsible parties. The user manages the progress, and the server reports the progress to all participants.

[0959] Example of a prompt:

[0960] "Please explain how to use an online meeting system that utilizes an emotion engine. This system automates everything from scheduling and meeting management to minute-taking and action plan management."

[0961] As described above, the system of the present invention streamlines and automates the entire process from preparation to conducting and follow-up of online meetings, enabling flexible meeting management that takes into account emotional states.

[0962] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0963] Step 1:

[0964] A user submits an online meeting request to the system. The request includes detailed information such as the meeting date and time, participant list, and agenda. This information is processed as user input data and sent to the server. The server receives it as an HTTP POST request and stores it in a database (MySQL). The stored data is organized into fields such as meeting ID, meeting date and time, participant list, and agenda.

[0965] Step 2:

[0966] The server communicates with the Google Calendar API to retrieve calendar information for all participants. The server uses each user's API key to send requests to the Google Calendar API, retrieving each participant's schedule and availability. The retrieved calendar information is returned to the server in JSON format. The server parses this JSON data and lists each participant's availability.

[0967] Step 3:

[0968] The server calculates the optimal date and time based on the calendar information it has obtained. Using an algorithm, it identifies the date and time when the largest number of participants are available. Specifically, it calculates the overlap of each participant's available time slots and extracts the time slot that matches the most participants. This calculation result is generated as a list of candidate dates and times.

[0969] Step 4:

[0970] The server notifies each user of the calculated date and time options. The server sends the generated list of date and time options to each user using the email system or the system's notification function. Users review this and approve or modify it through the system interface or mobile app. The final confirmed date and time is sent to the server based on the user's input and stored in the database as a confirmed date and time list.

[0971] Step 5:

[0972] The device collects materials related to the meeting agenda. Specifically, it uses the Google Drive API to retrieve necessary materials from a shared folder. The device collects metadata for each material (file name, creation date, size, etc.) and sends it to the server. The server organizes the received metadata and generates a list of materials.

[0973] Step 6:

[0974] The server organizes the collected materials and uploads them to a shared folder. The server then analyzes the list of materials, categorizes them by importance, and re-uploads them to Google Drive. This ensures that the materials are shared in a format accessible to all participants. Simultaneously, the server generates a meeting minutes template and distributes it to all participants in advance. A Python script is used to automatically generate the template, including slots for each agenda item.

[0975] Step 7:

[0976] The server sends reminder notifications one hour and ten minutes before the meeting starts. Firebase Cloud Messaging is used to send these reminder notifications to each participant's device. This ensures that users are prepared and don't forget the meeting start time.

[0977] Step 8:

[0978] The server provides a timekeeping function during meetings. Using a JavaScript library, it manages the scheduled time for each agenda item in real time. The server monitors the progress of each agenda item and displays the remaining time on the screen. This ensures smooth meeting progress.

[0979] Step 9:

[0980] The server presents relevant data in real time. It retrieves necessary data from external APIs and internal databases and displays it to participants during the meeting. This ensures that discussions are data-driven and decision-making is accelerated.

[0981] Step 10:

[0982] The emotion engine runs a deep learning model using TensorFlow to analyze the camera footage and audio of meeting participants. It recognizes emotional states in real time and classifies them as positive, negative, neutral, etc. This data is then sent to a server.

[0983] Step 11:

[0984] The server adjusts the meeting flow based on emotional data. If participants' concentration levels decline, the server sends a notification prompting them to take a break. This improves the effectiveness of the meeting.

[0985] Step 12:

[0986] The server generates meeting minutes based on emotional states. A Python script is used to incorporate emotional data into the minutes. This clarifies the emotional context of statements, which is useful for subsequent analysis.

[0987] Step 13:

[0988] The terminal records the meeting content and sends the data to the server. The recorded data is saved as an audio file. The server uses a TensorFlow speech recognition model to convert the recorded data into text and generate character data.

[0989] Step 14:

[0990] The server automatically generates meeting minutes based on the generated text data. It organizes the text according to a pre-distributed template to produce complete meeting minutes. The minutes are distributed to all participants via email and system notifications.

[0991] Step 15:

[0992] The server automatically extracts action plans from meeting minutes and assigns them to responsible personnel. Using natural language processing technology, it generates specific tasks from the meeting minutes and automatically assigns them to the appropriate individuals. This streamlines post-meeting task management.

[0993] Step 16:

[0994] Users manage and update the progress of their action plans on the system. Users input and update progress through the system's web interface, and this data is sent to the server in real time. The server periodically reports the progress to all participants. This ensures that everyone is aware of the latest status.

[0995] As described above, this system streamlines the entire online meeting process and enables flexible meeting management that takes into account emotional states.

[0996] (Application Example 2)

[0997] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0998] In modern brick-and-mortar stores, efficient and effective customer service is essential, but it is difficult for staff to grasp the emotional state and interests of individual customers in real time. Furthermore, proper scheduling and reminder functions are necessary to efficiently handle multiple customers simultaneously, but traditional tools lack sufficient integration of these functions, leading to decreased staff efficiency.

[0999] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1000] In this invention, the server includes means for receiving meeting requests, means for acquiring meeting participant schedule information and calculating the optimal date and time, means for collecting and sharing meeting-related materials and data, means for managing scheduled time for each agenda item during the meeting and presenting relevant data in real time, means for recording meeting content, automatically transcribing it into text to generate meeting minutes, means for automatically extracting implementation plans from meeting minutes and assigning them to responsible personnel, means for recognizing participants' emotional states in real time using an emotion analysis engine and optimizing the progress of the meeting using that data, and means for managing customer service schedules and providing reminder notifications. This makes it possible to grasp customers' emotional states in real time in physical stores and respond quickly to the optimal needs, and also enables scheduling and reminder functions for efficiently handling multiple customers.

[1001] "Means for receiving meeting requests" refers to a device or software for acquiring online meeting requests and processing that data within the system.

[1002] "Means for obtaining meeting participants' schedule information and calculating the optimal date and time" refers to a device or software that has the function of checking the participants' schedules and automatically calculating the optimal date and time when everyone can attend.

[1003] "Means for collecting and sharing materials and data related to a meeting" refers to devices or software that collect materials and data used in a meeting and make them accessible to all participants.

[1004] "A means of managing scheduled time for each agenda item during a meeting and presenting relevant data in real time" refers to a system that manages time in accordance with the progress of the meeting and displays necessary information in real time.

[1005] "Methods for recording meeting content, automatically transcribing it into text, and generating meeting minutes" refers to a device or software that records audio during a meeting, automatically converts that audio data into text, and organizes and saves it as meeting minutes.

[1006] "A method for automatically extracting implementation plans from meeting minutes and assigning them to responsible personnel" refers to a system that analyzes specific action plans from generated meeting minutes and automatically assigns them to individual responsible personnel.

[1007] "A means of recognizing participants' emotional states in real time using an emotion analysis engine and optimizing the progress of a meeting using that data" refers to a device or software that analyzes participants' emotions in real time and adjusts and optimizes the progress of a meeting based on that emotional data.

[1008] "A means of managing customer service schedules and sending reminder notifications" refers to a system that manages customer service schedules and automatically sends reminder notifications.

[1009] The system of the present invention is designed to streamline and effectively handle customer interactions in physical stores, and is particularly characterized by its use of an emotion analysis engine to recognize the customer's emotional state in real time and optimize responses based on that recognition. Specific embodiments of this system will be described in detail below.

[1010] First, the server has a means of receiving online meeting requests. This means the meeting request is retrieved, and the data is processed within the system. This request includes detailed information such as the date and time, participant list, and agenda. The server stores this information in a database for use in subsequent processing.

[1011] Next, the server has a means to obtain the schedule information of meeting participants and calculate the optimal date and time. Using this means, the server communicates with each participant's calendar API to obtain their schedule information. Based on the obtained information, it calculates the date and time when the largest number of participants are available and notifies the user of the candidate date and time.

[1012] Furthermore, the server has a means of collecting and sharing materials and data related to the meeting. Through this means, the server collects necessary materials from shared folders and external databases, organizes them, and uploads them to the shared folder in a format accessible to all participants.

[1013] During the meeting, the server manages the scheduled time for each agenda item and has a means to display relevant data in real time. Using this means, the server manages the time in accordance with the progress of the meeting and displays the necessary information in real time. This allows participants to access the necessary data immediately.

[1014] Furthermore, the server has a means to record meeting content, automatically transcribe it into text, and generate meeting minutes. This means the server records the audio during the meeting and automatically transcribes the audio data into text. The generated text is then organized into meeting minutes according to a pre-distributed template. The server also has a means to automatically extract action plans from the meeting minutes and assign them to the responsible parties. This means the server analyzes the action plans from the generated minutes and automatically assigns them to the responsible parties.

[1015] The system also includes a means to use an emotion analysis engine to recognize participants' emotional states in real time and use that data to optimize the meeting's progress. This means the server can acquire emotional data in real time and appropriately adjust the meeting's progress, for example, by prompting participants to take a break if their concentration is waning.

[1016] Finally, the server has a means of managing customer service schedules and sending reminder notifications. This allows the server to manage customer service appointments and automatically send reminder notifications, enabling staff to efficiently handle multiple customers.

[1017] As a specific example, when a staff member wearing smart glasses efficiently attends to multiple customers, the following prompt message will be displayed.

[1018] "Current customers want to learn more about the products they've shown interest in. Provide them with relevant information and, if necessary, offer customized suggestions."

[1019] This system allows businesses to understand customers' emotional states in real time at physical stores, enabling them to respond quickly and optimally, thereby improving the quality of customer service.

[1020] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1021] Step 1:

[1022] Input: The user submits a request for an online meeting.

[1023] Action: The server receives this request and saves its contents to the database. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1024] Output: Request information is saved in the database.

[1025] Step 2:

[1026] Input: The server accesses each user's calendar API to retrieve the meeting participants' schedule information.

[1027] Operation: The server retrieves the schedule information of each participant and uses that information to calculate the date and time when the largest number of participants are available.

[1028] Output: The server notifies the user of the list of optimal date and time options.

[1029] Step 3:

[1030] Input: The server notifies the user of possible dates and times, and the user confirms the final date and time.

[1031] Action: The date and time confirmed by the user will be finalized as the meeting date and time.

[1032] Output: The final date and time are confirmed and notified to meeting participants.

[1033] Step 4:

[1034] Input: The server accesses shared folders and external databases to collect materials and data related to the meeting agenda.

[1035] Operation: The server organizes the collected data and uploads it to a shared folder in a format accessible to all participants.

[1036] Output: Necessary documents and data are saved in a shared folder.

[1037] Step 5:

[1038] Input: At the start of the meeting, the server sends a reminder notification to all participants.

[1039] Function: Sends reminder notifications to multiple devices to encourage meeting preparations.

[1040] Output: A reminder notification will be sent to all participants before the meeting starts.

[1041] Step 6:

[1042] Input: The server manages the scheduled time for each agenda item during the meeting and presents relevant data in real time.

[1043] Function: Manages time according to the progress of the meeting and displays necessary information in real time.

[1044] Output: Participants can immediately access the data they need.

[1045] Step 7:

[1046] Input: The server records the audio during the meeting.

[1047] Operation: The server uses speech recognition technology to automatically convert recorded audio data into text.

[1048] Output: The transcribed meeting content is generated as meeting minutes.

[1049] Step 8:

[1050] Input: The server automatically generates meeting minutes based on the generated text.

[1051] Operation: The server automatically generates meeting minutes and organizes them according to a pre-distributed template.

[1052] Output: Meeting minutes will be distributed to all participants via email and in-system notifications.

[1053] Step 9:

[1054] Input: The server uses an emotion analysis engine to recognize the emotional state of participants in real time.

[1055] Operation: The server uses sentiment data to optimize the meeting's progress and adjust the flow as needed. For example, it might prompt participants to take a break if their concentration is waning.

[1056] Output: Optimized meeting management is achieved.

[1057] Step 10:

[1058] Input: The server automatically extracts the implementation plan from the meeting minutes.

[1059] Operation: The server analyzes the implementation plan and assigns it to the responsible person.

[1060] Output: The implementation plan is assigned to the person in charge.

[1061] Step 11:

[1062] Input: The server retrieves data for managing customer service schedules.

[1063] Operation: The server manages customer service schedules and automatically sends reminder notifications.

[1064] Output: A reminder notification will be sent at the specified time.

[1065] The specific software used in each step includes MySQL for database management, the Google Calendar API for communication with the calendar API, the Google Cloud Speech-to-Text API for speech recognition, DeepFace for sentiment analysis, and Python program code for controlling the system.

[1066] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1067] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1068] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1069] [Third Embodiment]

[1070] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1071] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1072] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1073] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1074] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1075] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1076] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1077] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1078] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1079] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1080] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1081] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1082] The system of this invention combines multiple functions to enable the efficient and automated operation of online meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[1083] 1. Accepting requests

[1084] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1085] The server receives this request and saves it to the database. This data is used for subsequent processing.

[1086] 2. Scheduling

[1087] The server communicates with each user's calendar API to retrieve the calendar information of all participants.

[1088] Based on the acquired calendar information, the server calculates the optimal date and time. For example, it lists dates and times when many participants are available.

[1089] These proposed dates and times will be notified to the users for each participant to review. The date and time will be finalized upon final approval.

[1090] 3. Meeting preparations

[1091] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[1092] The server organizes the collected data and uploads it to a shared folder.

[1093] Furthermore, the server creates meeting minutes templates based on the agenda items and distributes them to all participants.

[1094] 4. Meeting Management

[1095] The server sends a reminder notification to the user at the start of the meeting.

[1096] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data to the user's device in real time, depending on the agenda item.

[1097] 5. Recording of discussions and creation of meeting minutes.

[1098] The device records the meeting content and sends the data to the server.

[1099] The server converts the audio data into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[1100] 6. Action plan generation and management

[1101] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the responsible person.

[1102] Users manage the progress of their assigned action plans and update them through the system as needed.

[1103] The server will periodically report the progress of the action plan to all participants.

[1104] Specific example

[1105] For example, suppose a company wants to hold an online meeting for a new product development project. In this case, the user proposes the meeting regarding the new product development, enters the date and time, participant list, and agenda into the system, and submits the request.

[1106] The server receives the request and retrieves each participant's calendar information. It calculates the optimal date and time and notifies everyone. All participants review the proposed dates and times and finalize the date and time.

[1107] Before the meeting, terminals collect market research data and competitor analysis materials, and the server organizes and uploads them to a shared folder. Additionally, meeting minute templates tailored to the agenda are generated and distributed to participants in advance.

[1108] During the meeting, the server keeps track of the time and displays relevant data in real time for each agenda item. The meeting content is recorded by the terminals, and the server converts the audio data into text to automatically create meeting minutes.

[1109] After the meeting ends, the server extracts action plans from the meeting minutes and assigns them to the responsible parties. Users manage and update the progress of the action plans within the system, and the server reports the progress to all participants.

[1110] As a result, it becomes possible to streamline and automate the entire process, from meeting preparation to conducting and follow-up.

[1111] The following describes the processing flow.

[1112] Step 1:

[1113] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1114] Step 2:

[1115] The server receives this request and saves it to the database. This data is used for subsequent processing.

[1116] Step 3:

[1117] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[1118] Step 4:

[1119] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[1120] Step 5:

[1121] The server notifies users of a list of possible dates and times, and each participant confirms them. Based on the participants' confirmations, the optimal date and time are finalized.

[1122] Step 6:

[1123] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[1124] Step 7:

[1125] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[1126] Step 8:

[1127] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[1128] Step 9:

[1129] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[1130] Step 10:

[1131] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[1132] Step 11:

[1133] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[1134] Step 12:

[1135] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[1136] Step 13:

[1137] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[1138] Step 14:

[1139] The server automatically generates meeting minutes based on the generated text. The minutes are organized according to a template distributed in advance.

[1140] Step 15:

[1141] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[1142] Step 16:

[1143] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[1144] Step 17:

[1145] Manage the progress of action plans assigned to users. Update task progress on the system.

[1146] Step 18:

[1147] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[1148] The above is the specific processing flow of the program for this system.

[1149] (Example 1)

[1150] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1151] Current online meeting systems make it difficult to efficiently manage the entire process, from meeting preparation to execution and follow-up. Furthermore, tasks such as timekeeping, real-time presentation of relevant materials during meetings, and post-meeting minute-taking and action plan extraction and management are not automated, relying heavily on manual processes. This results in significant effort and time being required for effective meeting management.

[1152] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1153] In this invention, the server includes means for receiving meeting requests, means for obtaining the schedule information of meeting participants and calculating the optimal date and time, means for collecting and sharing materials and information related to the meeting, means for managing the scheduled time for each agenda item during the meeting and presenting relevant information in real time, means for recording the meeting content, automatically transcribing it into text to generate meeting minutes, means for automatically extracting action plans from the meeting minutes and assigning them to responsible persons, means for sending reminders at the start of the meeting, means for sending meeting recording data to the server, and means for periodically reporting the progress of the action plans. This makes it possible to streamline and automate the entire process from meeting preparation to execution and follow-up.

[1154] "Means for receiving meeting requests" refers to the means by which users propose online meetings and send detailed information such as the date and time, participant list, and agenda to the server.

[1155] "A means of obtaining the schedule information of meeting participants and calculating the optimal date and time" refers to a method in which the server communicates with each participant's calendar API and calculates the optimal date and time that all participants can attend based on the information obtained.

[1156] "Means for collecting and sharing materials and information related to a meeting" refers to a method by which terminals collect necessary materials for a meeting from shared folders or external databases, and a server organizes and shares them with participants.

[1157] "A means of managing scheduled time for each agenda item during a meeting and presenting relevant information in real time" refers to a method in which a server manages the scheduled time during the meeting and presents relevant materials and information in real time as needed.

[1158] "A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes" refers to a method in which a terminal records the audio during a meeting, and a server uses speech recognition technology to transcribe that data into text and compile it into meeting minutes.

[1159] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a method in which a server analyzes the text of the meeting minutes, extracts specific action items, and automatically assigns them to the appropriate personnel.

[1160] "Method for sending reminders at the start of a meeting" refers to a method by which the server sends a reminder to all participants to notify them of the start time as the meeting is about to begin.

[1161] "Means for sending meeting recording data to the server" refers to the means by which a terminal uploads the recorded meeting audio data to the server.

[1162] "Means for periodically reporting on the progress of the action plan" refers to a method by which the server periodically checks the implementation status of the extracted action plan and reports its progress to all participants.

[1163] The system of the present invention enables the efficient and automated operation of online meetings through the collaborative roles of the server, terminal, and user. Specific embodiments are described below.

[1164] Accepting requests

[1165] A user enters and submits an online meeting request. This includes details such as the date and time, participant list, and agenda. For example, suppose a user submits a request for "Meeting on October 15, 2023 at 2:00 PM, with participants A, B, and C, on the topic of new product development." The server receives this request and stores it in the database. This makes the meeting details available for subsequent processing.

[1166] Example prompt:

[1167] I would like to hold a meeting regarding new product development on October 15, 2023, at 2:00 PM. The participants are Person A, Person B, and Person C. The topic of discussion will be "New Product Development".

[1168] Schedule adjustment

[1169] The server accesses the calendar APIs of all participants (e.g., Google Calendar API, Outlook Calendar API) to retrieve calendar information. Based on this information, the server calculates the optimal date and time and notifies the users of the suggested dates and times. Each user who receives the notification reviews the suggested dates and times and accepts the optimal one. Through this confirmation, the final date and time are determined.

[1170] Example prompt:

[1171] Please obtain each participant's calendar information, calculate a date and time that works for everyone, and notify them accordingly.

[1172] Meeting preparations

[1173] The terminal collects materials and data related to the meeting. These are collected from shared folders (e.g., Google Drive, Dropbox) or external databases (e.g., a company's internal database). The server organizes these materials and uploads them to the shared folder, as well as generating meeting minutes templates based on the agenda and distributing them to participants in advance.

[1174] Example prompt:

[1175] Collect market research data and competitive analysis materials, upload them to a shared folder, and generate a meeting minutes template to distribute to participants.

[1176] Meeting progress

[1177] The server sends a reminder notification at the start of the meeting. During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data related to the agenda item on the user's device in real time. For example, during a discussion on new product development, relevant market research data is automatically displayed.

[1178] Example prompt:

[1179] Send reminder notifications at the start of the meeting, manage the scheduled time for each agenda item, and present relevant data in real time.

[1180] Recording of discussions and creation of meeting minutes

[1181] The terminal records the meeting content and sends the data to the server. The server uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the recording into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[1182] Example prompt:

[1183] Please record the meeting, transcribe the audio data into text, and automatically create and distribute meeting minutes.

[1184] Action plan generation and management

[1185] The server extracts action plans from meeting minutes and automatically assigns them to the appropriate personnel. Natural language processing technology is used to identify important tasks and assign them to the correct individuals. Each user manages and updates the progress of their assigned action plan within the system. The server periodically reports the progress to all participants.

[1186] Example prompt:

[1187] Extract action plans from the meeting minutes, assign them to the responsible parties, and have them report on the progress regularly.

[1188] As described above, the system of the present invention can streamline and automate the entire process from preparation to implementation and follow-up of online meetings. Through this system, the effort and time required for meeting management can be significantly reduced.

[1189] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1190] Step 1:

[1191] The user enters and submits an online meeting request. The user fills in details such as the date and time, participant list, and agenda in the form and presses the "Submit" button. The input data includes the meeting date and time, participant list, and agenda.

[1192] Specific actions:

[1193] The user opens a web form.

[1194] Fill in the details of the meeting in the input field.

[1195] Click the "Send" button.

[1196] Output: Meeting request data received by the server.

[1197] Step 2:

[1198] The server receives requests from users and stores them in the database. The server receives HTTP POST requests, parses their contents, and adds new records to the database.

[1199] Specific actions:

[1200] The server waits for HTTP requests.

[1201] User request received.

[1202] The request content is analyzed and saved to the "Meeting Requests" table.

[1203] Input: Meeting request data from the user.

[1204] Output: Meeting request information stored in the database.

[1205] Step 3:

[1206] The server communicates with all participants' calendar APIs to retrieve calendar information. The server uses each participant's authentication token to access the calendar API and retrieve event information.

[1207] Specific actions:

[1208] The server sends a request to the calendar API.

[1209] Retrieve the calendar information of each participant.

[1210] Analyze the calendar information to list available dates and times.

[1211] Input: Participant authentication token.

[1212] Output: Participant calendar information.

[1213] Step 4:

[1214] Based on the calendar information obtained by the server, the optimal date and time for all participants to attend is calculated. These suggested dates and times are then notified to the users.

[1215] Specific actions:

[1216] Analyze the acquired calendar information.

[1217] The algorithm calculates the dates and times when everyone is free.

[1218] The notification sending function will send the user a list of possible dates and times.

[1219] Input: Participant's calendar information.

[1220] Output: A list of possible dates and times sent to the user.

[1221] Step 5:

[1222] The user reviews the suggested dates and times and approves the most suitable one. Upon receiving the notification, the user selects the appropriate date and time from the list of options and replies to the system.

[1223] Specific actions:

[1224] The user opens the notification.

[1225] Check the suggested dates and times.

[1226] Select a date and time for approval and reply to the system.

[1227] Input: A list of possible dates and times from the server.

[1228] Output: Final date and time of approval.

[1229] Step 6:

[1230] The device collects meeting-related materials and data. It accesses shared folders (e.g., Google Drive, Dropbox) and external databases to retrieve and organize necessary materials.

[1231] Specific actions:

[1232] The device accesses shared folders and external databases.

[1233] Download the necessary documents.

[1234] Classify and organize the documents.

[1235] Input: Access information for data collection.

[1236] Output: Collected and organized materials.

[1237] Step 7:

[1238] The server uploads the collected materials to a shared folder, creates a meeting minutes template based on the agenda, and distributes it to the participants.

[1239] Specific actions:

[1240] The server uploads the documents to the shared folder.

[1241] Create a meeting minutes template using the template generation module.

[1242] Distribute the generated template to participants via email or shared link.

[1243] Input: Collected and organized materials.

[1244] Output: Documents uploaded to the shared folder, meeting minutes templates distributed to participants.

[1245] Step 8:

[1246] The server will send a reminder notification when the meeting starts.

[1247] Specific actions:

[1248] The server executes code that sends a reminder 5 minutes before the start time.

[1249] A reminder notification will be sent to each participant's device.

[1250] Input: Meeting start time information.

[1251] Output: Reminder notifications sent to participants.

[1252] Step 9:

[1253] The server manages the scheduled meeting time and keeps track of it. During the meeting, it displays relevant data to the user's terminal in real time, depending on the agenda.

[1254] Specific actions:

[1255] The server sets a timer and manages the scheduled time.

[1256] Timekeeping alerts are sent to each user's device.

[1257] Relevant data will be delivered in real time.

[1258] Input: Scheduled time information for each agenda item.

[1259] Output: Real-time data displayed on the user's device.

[1260] Step 10:

[1261] The device records the meeting content and sends the data to the server.

[1262] Specific actions:

[1263] The device launches the voice recording app.

[1264] Record the meeting content and save it to local storage.

[1265] Upload the recording file to the server.

[1266] Input: Audio data from a meeting.

[1267] Output: Recording file uploaded to the server.

[1268] Step 11:

[1269] The server uses a speech recognition API to convert the audio data into text and generate meeting minutes.

[1270] Specific actions:

[1271] The server sends audio data to the speech recognition API.

[1272] The acquired text data is formatted and compiled into meeting minutes.

[1273] Input: Audio file.

[1274] Output: Generated meeting minutes.

[1275] Step 12:

[1276] The server extracts action plans from meeting minutes and automatically assigns them to responsible personnel. The server uses natural language processing technology to identify important tasks and notifies the responsible personnel.

[1277] Specific actions:

[1278] The server analyzes the meeting minutes and extracts the tasks.

[1279] Assign the extracted tasks to the responsible personnel.

[1280] Input: Generated meeting minutes.

[1281] Output: Action plan assigned to the person in charge.

[1282] Step 13:

[1283] Users manage the progress of their assigned action plans and update them through the system as needed.

[1284] Specific actions:

[1285] The user logs into the system.

[1286] Use the interface to update the progress of the action plan.

[1287] Update the progress status.

[1288] Input: Progress status of the action plan.

[1289] Output: Updated progress.

[1290] Step 14:

[1291] The server will periodically report the progress of the action plan to all participants.

[1292] Specific actions:

[1293] The server checks the progress at regular intervals.

[1294] Summarize the progress and generate a report.

[1295] Send the report to all participants.

[1296] Input: Updated progress status.

[1297] Output: Progress report sent to all participants.

[1298] (Application Example 1)

[1299] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1300] Traditional online meeting systems faced challenges such as requiring significant time and effort for meeting preparation, execution, and follow-up. Furthermore, recording meetings and creating minutes were cumbersome, making efficient operation difficult. This was especially true when multiple participants were involved, as smoothly managing all processes—scheduling, organizing materials, automatically generating minutes, and managing action plans—became even more challenging. As a result, the quality of meetings often suffered, hindering the maximization of results.

[1301] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1302] In this invention, the server includes means for receiving meeting requests, means for obtaining calendar information of meeting participants and calculating the optimal date and time, means for collecting meeting-related materials and data, organizing them in a shared folder and uploading them, means for managing scheduled times for each agenda item and presenting relevant data to the user terminal in real time, means for recording meeting content, automatically transcribing it into text and generating meeting minutes, means for automatically extracting action plans from meeting minutes and assigning them to responsible persons, means for communicating with participants' calendars to calculate the optimal date and time and notifying the user, means for collecting and organizing relevant data and sharing a webinar page automatically created using a template, means for providing relevant information in real time during the meeting, and means for inputting prompt sentences using a generation AI model and transcribing audio data into text. This makes it possible to streamline and automate the entire process from meeting preparation to progress and follow-up.

[1303] "Means of receiving meeting requests" refers to a function where a user proposes an online meeting, sends detailed information such as the date and time, participant list, and agenda to the system, and the server receives the request.

[1304] The "method for obtaining meeting participants' calendar information and calculating the optimal date and time" is a function that communicates with each meeting participant's calendar API, checks everyone's availability, and calculates the most suitable date and time for the meeting.

[1305] "A means of collecting meeting-related materials and data, organizing them in a shared folder, and uploading them" refers to a function that collects necessary materials and data from different sources, organizes them, and uploads them to a single shared folder.

[1306] "A means of managing scheduled time for each agenda item and presenting relevant data to the user's terminal in real time" refers to a function that manages the scheduled time allocated to each agenda item and displays relevant data on the user's terminal in real time during the meeting.

[1307] "A method for recording meeting content, automatically converting it to text, and generating meeting minutes" refers to a function that records audio during a meeting, automatically converts that audio data into text, and generates meeting minutes.

[1308] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a function that extracts specific action plans from generated meeting minutes and automatically assigns them to the responsible personnel.

[1309] "A means of communicating with participants' calendars to calculate the optimal date and time and notifying the user" refers to a function that communicates with meeting participants' calendar information, calculates multiple date and time options, and notifies the user.

[1310] "A means of collecting and organizing relevant data and sharing a webinar page automatically created using a template" refers to a function that collects the necessary data related to a webinar, organizes it, and then automatically generates and shares a webinar page using a template.

[1311] "Means of providing relevant information in real time during a meeting" refers to a function that provides information and materials related to the agenda in real time while the meeting is in progress.

[1312] "A means of inputting prompt sentences using a generative AI model and converting audio data to text" refers to a function that utilizes a generative AI model to input prompt sentences for converting audio data to text, and then converts the audio data to text based on these prompts.

[1313] The present invention provides a multi-functional system that enables the efficient and automated operation of online meetings. In this system, the server, terminals, and users each play specific roles and work together.

[1314] System Configuration

[1315] hardware

[1316] Smartphones (iOS / Android): Devices used by users and meeting participants.

[1317] Server: A central computer that processes and manages data.

[1318] Internet connection: Responsible for communication between all devices.

[1319] software

[1320] Client-side: React Native

[1321] Server-side: Node.js, Express

[1322] Database: MongoDB

[1323] AI speech recognition: Google Cloud Speech-to-Text API

[1324] Calendar APIs: Google Calendar API, Microsoft Outlook API

[1325] Function flow

[1326] 1. Means of receiving meeting requests

[1327] A user sends an online meeting request to the server using a smartphone application. This request includes detailed information such as the date and time of the meeting, a list of participants, and the agenda. The server receives this information and stores it in its database.

[1328] 2. A means of obtaining the calendar information of meeting participants and calculating the optimal date and time.

[1329] The server communicates with each meeting participant's calendar API (such as Google Calendar API or Microsoft Outlook API) to retrieve everyone's availability. Based on the retrieved calendar information, the server calculates the optimal date and time and notifies the user.

[1330] 3. A method for collecting meeting-related materials and data, organizing them in a shared folder, and uploading them.

[1331] The user's device collects the necessary materials and data for the meeting, and the server organizes them and uploads them to a shared folder. This shared folder is accessible to the user and meeting participants.

[1332] 4. A means of managing the scheduled time for each agenda item and presenting relevant data to the user's terminal in real time.

[1333] The server manages the scheduled time for each agenda item during the meeting and tracks the progress of time. It also displays relevant data to the user's smartphone in real time.

[1334] 5. A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes.

[1335] The user's smartphone records the meeting and sends the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio data into text and automatically generates meeting minutes.

[1336] 6. A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel.

[1337] The server extracts action plans from the generated meeting minutes and assigns them to responsible parties. Users then manage the progress of these action plans based on these plans.

[1338] 7. Means of providing relevant information in real time

[1339] The server provides users with relevant information and materials in real time during the meeting, based on the agenda. This feature ensures that the meeting proceeds smoothly.

[1340] 8. A method for inputting prompt text using a generative AI model and converting speech data into text.

[1341] Users can leverage a generative AI model to input appropriate prompts and efficiently convert audio data into text.

[1342] Specific example

[1343] When a company holds an online webinar to announce a new product, this system automates and streamlines all preparation and execution. For example, a user submits a request for a new product announcement webinar to the system, and the server retrieves each participant's calendar information to calculate the optimal date and time. Relevant materials are collected, and the webinar page is automatically generated based on an organized template. During the meeting, relevant information is provided in real time, and after the meeting, the audio data is transcribed into text to generate meeting minutes.

[1344] Examples of prompts to input into a generative AI model:

[1345] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new product features, market strategy, and Q&A after the session. The audio recording is available below."

[1346] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1347] Step 1:

[1348] A user sends an online meeting request via a smartphone app. The user enters detailed information such as the meeting date and time, participant list, and agenda. This information is then sent to a server and stored in a database. The database contains fields for meeting ID, date and time, participant list, and agenda. Specifically, the user fills in the required information in the app's input form and presses the submit button.

[1349] Step 2:

[1350] The server retrieves participants' calendar information based on meeting information stored in the database. As input, the server sends requests to each participant's calendar API (such as Google Calendar API or Microsoft Outlook API) based on the meeting ID and participant list. As output, the server receives each participant's free time. Specifically, the server retrieves each participant's calendar information in parallel and compiles everyone's free time into a list.

[1351] Step 3:

[1352] The server calculates the optimal meeting date and time based on the retrieved calendar information. It receives a list of participants' available time slots as input. As output, it generates a list of the optimal dates and times when all participants are available. Specifically, the server compares the available time slot lists and executes logic to select the date and time when the most participants are available.

[1353] Step 4:

[1354] The server notifies the user of the calculated optimal date and time. As input, it generates a notification message based on a list of optimal dates and times. As output, this notification message is sent to the user's smartphone, making it available for the user to view. Specifically, the server sends the message to the smartphone via a push notification API.

[1355] Step 5:

[1356] The terminal collects the necessary materials and data for the meeting and sends them to the server. As input, it receives access links to materials provided by users and meeting participants, as well as uploaded files. As output, the materials and data are uploaded to the server and organized in a shared folder. Specifically, users use the app's material upload function to select and submit the necessary files.

[1357] Step 6:

[1358] The server organizes and uploads received documents and data to a shared folder. It receives collected documents and data as input. As output, these are categorized and saved in a designated folder, making them accessible to participants. Specifically, the server automatically generates a directory structure and appropriately places the documents.

[1359] Step 7:

[1360] The server manages the scheduled time and agenda during a meeting. It receives the start time and the scheduled time for each agenda item as input. Its output includes real-time progress tracking and displays warnings as the scheduled end time approaches. Specifically, the server uses a timer function to monitor time and issues alerts each time the scheduled time is reached.

[1361] Step 8:

[1362] The server records the meeting content and sends the audio data to the Google Cloud Speech-to-Text API for transcription. It receives the recorded audio data as input and automatically generated text data as output. Specifically, the server sends the audio file to the API and converts the returned text data into meeting minutes.

[1363] Step 9:

[1364] The server extracts action plans from meeting minutes and assigns them to responsible personnel. It analyzes the transcribed meeting minutes as input. The output determines each action item and its assigned person. Specifically, the server uses natural language processing to analyze the meeting minutes and automatically generates and assigns action plans.

[1365] Step 10:

[1366] The server uses a generative AI model to take prompt text as input and provides a means to convert audio data into text. As input, audio data is sent to the model along with the prompt text. As output, text data based on the prompt is generated. Specifically, the server appropriately constructs the prompt text and performs immediate analysis on the AI ​​model.

[1367] Examples of prompts to input into a generative AI model:

[1368] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new product features, market strategy, and Q&A after the session. The audio recording is available below."

[1369] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1370] The system of the present invention provides a combination of multiple functions for efficiently and highly automating online meetings. In particular, the quality of meetings can be further improved by incorporating an emotion engine. This system is centered around a server, terminals, and users, each playing a specific role.

[1371] 1. Accepting requests

[1372] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1373] The server receives the request and saves it to the database. This data is used for subsequent processing.

[1374] 2. Scheduling

[1375] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[1376] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[1377] These proposed dates and times will be notified to the users for each participant to review. The date and time will be finalized upon final approval.

[1378] 3. Meeting preparations

[1379] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[1380] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[1381] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[1382] 4. Meeting Management

[1383] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[1384] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[1385] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[1386] 5. Utilizing the Emotion Engine

[1387] The emotion engine analyzes camera footage and audio data from meeting participants to recognize each participant's emotional state in real time.

[1388] The server adjusts the meeting's progress based on emotional data obtained from the emotion engine. For example, if a participant's concentration is waning, it can send a notification prompting them to take a break.

[1389] The server incorporates emotional data into the meeting minutes, recording the emotional state in relation to the topic or statement in question.

[1390] 6. Recording of discussions and creation of meeting minutes.

[1391] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[1392] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[1393] The server automatically generates meeting minutes based on the generated text. The minutes are organized according to a pre-distributed template. Sentimental data is also recorded, making it easier to understand the emotional context of the statements.

[1394] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[1395] 7. Action plan generation and management

[1396] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[1397] Users manage the progress of their assigned action plans. They update the task progress on the system.

[1398] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[1399] Specific example

[1400] For example, suppose a company wants to hold an online meeting for a new product development project. In this case, the user proposes the meeting regarding the new product development, enters the date and time, participant list, and agenda into the system, and submits the request.

[1401] The server receives the request and retrieves each participant's calendar information. It calculates the optimal date and time and notifies everyone. All participants review the proposed dates and times and finalize the date and time.

[1402] Before the meeting, terminals collect market research data and competitor analysis materials, and the server organizes and uploads them to a shared folder. Additionally, meeting minute templates tailored to the agenda are generated and distributed to participants in advance.

[1403] During the meeting, the server keeps track of the time and displays relevant data in real time for each agenda item. An emotion engine analyzes the emotional state of participants and sends timely notifications for breaks or adjustments to the meeting's progress. The meeting content is recorded by the terminals, and the server converts the audio data into text to automatically create meeting minutes. Emotional data is also recorded in the minutes.

[1404] After the meeting ends, the server extracts action plans from the meeting minutes and assigns them to the responsible parties. Users manage and update the progress of the action plans within the system, and the server reports the progress to all participants.

[1405] As a result, it becomes possible to streamline and automate the entire process from meeting preparation to execution and follow-up, and to improve the quality of meetings by utilizing sentiment data.

[1406] The following describes the processing flow.

[1407] Step 1:

[1408] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1409] Step 2:

[1410] The server receives this request and saves it to the database. This data is used for subsequent processing.

[1411] Step 3:

[1412] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[1413] Step 4:

[1414] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[1415] Step 5:

[1416] The server notifies users of a list of possible dates and times, and each participant confirms them. Based on the participants' confirmations, the optimal date and time are finalized.

[1417] Step 6:

[1418] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[1419] Step 7:

[1420] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[1421] Step 8:

[1422] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[1423] Step 9:

[1424] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[1425] Step 10:

[1426] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[1427] Step 11:

[1428] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[1429] Step 12:

[1430] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[1431] Step 13:

[1432] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[1433] Step 14:

[1434] The emotion engine analyzes camera footage and audio data from meeting participants to recognize each participant's emotional state in real time.

[1435] Step 15:

[1436] The server adjusts the meeting's progress based on emotional data obtained from the emotion engine. For example, if a participant's concentration is waning, it can send a notification prompting them to take a break.

[1437] Step 16:

[1438] The server uses emotion data to reflect the emotional state of the participants in the meeting minutes.

[1439] Step 17:

[1440] The server creates meeting minutes based on automatically generated text. The minutes, organized according to a pre-distributed template, also reflect sentiment data.

[1441] Step 18:

[1442] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[1443] Step 19:

[1444] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[1445] Step 20:

[1446] Manage the progress of action plans assigned to users. Update task progress on the system.

[1447] Step 21:

[1448] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[1449] The above is the specific processing flow of the program for this system.

[1450] (Example 2)

[1451] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1452] In recent years, online meetings have become commonplace, but they require a great deal of manual intervention, and efficient meeting management and follow-up are essential. In particular, coordinating schedules for busy participants, sharing materials, creating meeting minutes, and managing participants' emotional states are all complex and time-consuming tasks. Managing post-meeting action plans and reporting progress also presents challenges. A system is needed to address these issues and streamline and enhance online meetings.

[1453] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving meeting requests, means for acquiring time management information of meeting participants and calculating the optimal date and time, means for collecting and sharing materials and data related to the meeting, means for managing the scheduled time for each agenda item during the meeting and presenting relevant data in real time, means for recording the meeting content, automatically converting it into text data to generate meeting minutes, means for automatically extracting action plans from the meeting minutes and assigning them to the responsible persons, means for recognizing the emotional state of meeting participants using a recording device and adjusting the progress of the meeting, and means for reflecting the emotional state of participants in response to their statements in the meeting minutes. This streamlines and automates the entire process from preparation to progress and follow-up of online meetings, enabling flexible meeting management that takes emotional states into consideration.

[1454] "Means for receiving meeting requests" refers to a function that receives requests from users that include detailed information such as the date and time of the meeting, participant list, and agenda, and stores this information in a database.

[1455] "Means for obtaining time management information of meeting participants" refers to a function that obtains the calendar information of meeting participants and allows for understanding each participant's schedule and available time.

[1456] The "means for calculating the optimal date and time" refers to a function that executes an algorithm based on acquired time management information to calculate the date and time when the largest number of participants are available.

[1457] "Means for collecting and sharing meeting-related materials and data" refers to the function of collecting materials and data necessary for the meeting agenda from external and internal databases and saving them in a shared folder in a format accessible to all participants.

[1458] "A means of managing scheduled time for each agenda item and presenting relevant data in real time" refers to a function that manages the scheduled time for each agenda item during a meeting and displays relevant data to participants in real time according to the progress.

[1459] "A method for recording meeting content and automatically converting it into text data to generate meeting minutes" refers to a function that records meeting content, automatically converts the recorded data into text data using speech recognition technology, and generates meeting minutes.

[1460] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a function that automatically extracts specific tasks from the content of the generated meeting minutes and assigns each task to the appropriate person.

[1461] "A means of recognizing the emotional state of meeting participants using a recording device and adjusting the progress of the meeting" refers to a function that analyzes the camera footage and audio of meeting participants, recognizes their emotional state in real time, and adjusts the progress of the meeting in a timely manner based on that data.

[1462] "Methods for reflecting emotional states in meeting minutes" refer to a function that records emotional data related to the content of statements in the minutes, thereby clarifying the emotional background of the statements.

[1463] The system of this invention provides a combination of multiple functions to efficiently and highly automate online meetings. The server, terminal, and user each play a specific role.

[1464] The server operates using the following software: MySQL for database management, Apache as the web server, and TensorFlow for speech recognition and sentiment analysis. The terminal uses the Google Drive API and built-in microphone for collecting documents and recording meeting content. Users operate the system through its web interface, but use the Google Calendar API to retrieve calendar information.

[1465] Accepting requests

[1466] A user submits an online meeting request to the system. The request includes detailed information such as the date and time of the meeting, a list of participants, and the agenda. The server receives the request and stores it in a database (MySQL). This allows the request information to be centrally managed within the system and used for subsequent processing.

[1467] Schedule adjustment

[1468] The server communicates with the Google Calendar API to retrieve calendar information for all participants. Based on the retrieved calendar information, an algorithm calculates the optimal date and time. The server notifies each user of the calculated date and time candidates, and the final date and time are confirmed once the users review and approve the candidates. This allows for efficient scheduling among busy participants.

[1469] Meeting preparations

[1470] The terminal collects materials related to the meeting agenda via the Google Drive API. The server organizes the collected materials in a Docker container and uploads them to Google Drive. This ensures that the materials are shared in a format accessible to all participants. Furthermore, the server generates a meeting minutes template using a Python script and distributes it to participants.

[1471] Meeting progress

[1472] The server sends reminder notifications using Firebase Cloud Messaging one hour and ten minutes before the start of the meeting. During the meeting, a JavaScript library is used for timekeeping, managing the scheduled time for each agenda item. In addition, relevant data is retrieved in real time from external APIs and internal databases and presented to participants. This streamlines the meeting process.

[1473] Using an Emotion Engine

[1474] The emotion engine analyzes participants' camera footage and audio, and uses TensorFlow to recognize their emotional state in real time. The server adjusts the meeting flow based on the emotion data and sends notifications prompting breaks if concentration levels decline. Furthermore, the emotion data is incorporated into the meeting minutes using a Python script.

[1475] Recording of discussions and creation of meeting minutes

[1476] The terminal records the meeting content and sends it to the server. The server uses a TensorFlow speech recognition model to transcribe the received recording into text and automatically creates meeting minutes. The minutes are distributed to all participants via email or in-system notifications. This ensures that the meeting content is recorded accurately and quickly.

[1477] Action plan generation and management

[1478] The server uses natural language processing technology to analyze meeting minutes and assign action plans to responsible personnel. Users manage and update progress on the system, and the server periodically reports the progress to all participants. This entire process streamlines post-meeting follow-up.

[1479] Examples of specific cases and prompt statements

[1480] When a company holds an online meeting for a new product development project, the user enters the date, time, participant list, and agenda into the system and sends a request. The server receives the request, retrieves each participant's calendar information, calculates the optimal date and time, and notifies them. All participants review the candidate dates and times and confirm the final date and time. Before the meeting, the terminal collects materials, the server organizes them, and uploads them to a shared folder. During the meeting, the server keeps track of the time and presents relevant data in real time. An emotion engine analyzes emotional states and prompts breaks at appropriate times. The meeting content is recorded by the terminal, and the server transcribes the audio data into text to create meeting minutes. After the meeting, the server extracts action plans from the minutes and assigns them to the responsible parties. The user manages the progress, and the server reports the progress to all participants.

[1481] Example of a prompt:

[1482] "Please explain how to use an online meeting system that utilizes an emotion engine. This system automates everything from scheduling and meeting management to minute-taking and action plan management."

[1483] As described above, the system of the present invention streamlines and automates the entire process from preparation to conducting and follow-up of online meetings, enabling flexible meeting management that takes into account emotional states.

[1484] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1485] Step 1:

[1486] A user submits an online meeting request to the system. The request includes detailed information such as the meeting date and time, participant list, and agenda. This information is processed as user input data and sent to the server. The server receives it as an HTTP POST request and stores it in a database (MySQL). The stored data is organized into fields such as meeting ID, meeting date and time, participant list, and agenda.

[1487] Step 2:

[1488] The server communicates with the Google Calendar API to retrieve calendar information for all participants. The server uses each user's API key to send requests to the Google Calendar API, retrieving each participant's schedule and availability. The retrieved calendar information is returned to the server in JSON format. The server parses this JSON data and lists each participant's availability.

[1489] Step 3:

[1490] The server calculates the optimal date and time based on the calendar information it has obtained. Using an algorithm, it identifies the date and time when the largest number of participants are available. Specifically, it calculates the overlap of each participant's available time slots and extracts the time slot that matches the most participants. This calculation result is generated as a list of candidate dates and times.

[1491] Step 4:

[1492] The server notifies each user of the calculated date and time options. The server sends the generated list of date and time options to each user using the email system or the system's notification function. Users review this and approve or modify it through the system interface or mobile app. The final confirmed date and time is sent to the server based on the user's input and stored in the database as a confirmed date and time list.

[1493] Step 5:

[1494] The device collects materials related to the meeting agenda. Specifically, it uses the Google Drive API to retrieve necessary materials from a shared folder. The device collects metadata for each material (file name, creation date, size, etc.) and sends it to the server. The server organizes the received metadata and generates a list of materials.

[1495] Step 6:

[1496] The server organizes the collected materials and uploads them to a shared folder. The server then analyzes the list of materials, categorizes them by importance, and re-uploads them to Google Drive. This ensures that the materials are shared in a format accessible to all participants. Simultaneously, the server generates a meeting minutes template and distributes it to all participants in advance. A Python script is used to automatically generate the template, including slots for each agenda item.

[1497] Step 7:

[1498] The server sends reminder notifications one hour and ten minutes before the meeting starts. Firebase Cloud Messaging is used to send these reminder notifications to each participant's device. This ensures that users are prepared and don't forget the meeting start time.

[1499] Step 8:

[1500] The server provides a timekeeping function during meetings. Using a JavaScript library, it manages the scheduled time for each agenda item in real time. The server monitors the progress of each agenda item and displays the remaining time on the screen. This ensures smooth meeting progress.

[1501] Step 9:

[1502] The server presents relevant data in real time. It retrieves necessary data from external APIs and internal databases and displays it to participants during the meeting. This ensures that discussions are data-driven and decision-making is accelerated.

[1503] Step 10:

[1504] The emotion engine runs a deep learning model using TensorFlow to analyze the camera footage and audio of meeting participants. It recognizes emotional states in real time and classifies them as positive, negative, neutral, etc. This data is then sent to a server.

[1505] Step 11:

[1506] The server adjusts the meeting flow based on emotional data. If participants' concentration levels decline, the server sends a notification prompting them to take a break. This improves the effectiveness of the meeting.

[1507] Step 12:

[1508] The server generates meeting minutes based on emotional states. A Python script is used to incorporate emotional data into the minutes. This clarifies the emotional context of statements, which is useful for subsequent analysis.

[1509] Step 13:

[1510] The terminal records the meeting content and sends the data to the server. The recorded data is saved as an audio file. The server uses a TensorFlow speech recognition model to convert the recorded data into text and generate character data.

[1511] Step 14:

[1512] The server automatically generates meeting minutes based on the generated text data. It organizes the text according to a pre-distributed template to produce complete meeting minutes. The minutes are distributed to all participants via email and system notifications.

[1513] Step 15:

[1514] The server automatically extracts action plans from meeting minutes and assigns them to responsible personnel. Using natural language processing technology, it generates specific tasks from the meeting minutes and automatically assigns them to the appropriate individuals. This streamlines post-meeting task management.

[1515] Step 16:

[1516] Users manage and update the progress of their action plans on the system. Users input and update progress through the system's web interface, and this data is sent to the server in real time. The server periodically reports the progress to all participants. This ensures that everyone is aware of the latest status.

[1517] As described above, this system streamlines the entire online meeting process and enables flexible meeting management that takes into account emotional states.

[1518] (Application Example 2)

[1519] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1520] In modern brick-and-mortar stores, efficient and effective customer service is essential, but it is difficult for staff to grasp the emotional state and interests of individual customers in real time. Furthermore, proper scheduling and reminder functions are necessary to efficiently handle multiple customers simultaneously, but traditional tools lack sufficient integration of these functions, leading to decreased staff efficiency.

[1521] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1522] In this invention, the server includes means for receiving meeting requests, means for acquiring meeting participant schedule information and calculating the optimal date and time, means for collecting and sharing meeting-related materials and data, means for managing scheduled time for each agenda item during the meeting and presenting relevant data in real time, means for recording meeting content, automatically transcribing it into text to generate meeting minutes, means for automatically extracting implementation plans from meeting minutes and assigning them to responsible personnel, means for recognizing participants' emotional states in real time using an emotion analysis engine and optimizing the progress of the meeting using that data, and means for managing customer service schedules and providing reminder notifications. This makes it possible to grasp customers' emotional states in real time in physical stores and respond quickly to the optimal needs, and also enables scheduling and reminder functions for efficiently handling multiple customers.

[1523] "Means for receiving meeting requests" refers to a device or software for acquiring online meeting requests and processing that data within the system.

[1524] "Means for obtaining meeting participants' schedule information and calculating the optimal date and time" refers to a device or software that has the function of checking the participants' schedules and automatically calculating the optimal date and time when everyone can attend.

[1525] "Means for collecting and sharing materials and data related to a meeting" refers to devices or software that collect materials and data used in a meeting and make them accessible to all participants.

[1526] "A means of managing scheduled time for each agenda item during a meeting and presenting relevant data in real time" refers to a system that manages time in accordance with the progress of the meeting and displays necessary information in real time.

[1527] "Methods for recording meeting content, automatically transcribing it into text, and generating meeting minutes" refers to a device or software that records audio during a meeting, automatically converts that audio data into text, and organizes and saves it as meeting minutes.

[1528] "A method for automatically extracting implementation plans from meeting minutes and assigning them to responsible personnel" refers to a system that analyzes specific action plans from generated meeting minutes and automatically assigns them to individual responsible personnel.

[1529] "A means of recognizing participants' emotional states in real time using an emotion analysis engine and optimizing the progress of a meeting using that data" refers to a device or software that analyzes participants' emotions in real time and adjusts and optimizes the progress of a meeting based on that emotional data.

[1530] "A means of managing customer service schedules and sending reminder notifications" refers to a system that manages customer service schedules and automatically sends reminder notifications.

[1531] The system of the present invention is designed to streamline and effectively handle customer interactions in physical stores, and is particularly characterized by its use of an emotion analysis engine to recognize the customer's emotional state in real time and optimize responses based on that recognition. Specific embodiments of this system will be described in detail below.

[1532] First, the server has a means of receiving online meeting requests. This means the meeting request is retrieved, and the data is processed within the system. This request includes detailed information such as the date and time, participant list, and agenda. The server stores this information in a database for use in subsequent processing.

[1533] Next, the server has a means to obtain the schedule information of meeting participants and calculate the optimal date and time. Using this means, the server communicates with each participant's calendar API to obtain their schedule information. Based on the obtained information, it calculates the date and time when the largest number of participants are available and notifies the user of the candidate date and time.

[1534] Furthermore, the server has a means of collecting and sharing materials and data related to the meeting. Through this means, the server collects necessary materials from shared folders and external databases, organizes them, and uploads them to the shared folder in a format accessible to all participants.

[1535] During the meeting, the server manages the scheduled time for each agenda item and has a means to display relevant data in real time. Using this means, the server manages the time in accordance with the progress of the meeting and displays the necessary information in real time. This allows participants to access the necessary data immediately.

[1536] Furthermore, the server has a means to record meeting content, automatically transcribe it into text, and generate meeting minutes. This means the server records the audio during the meeting and automatically transcribes the audio data into text. The generated text is then organized into meeting minutes according to a pre-distributed template. The server also has a means to automatically extract action plans from the meeting minutes and assign them to the responsible parties. This means the server analyzes the action plans from the generated minutes and automatically assigns them to the responsible parties.

[1537] The system also includes a means to use an emotion analysis engine to recognize participants' emotional states in real time and use that data to optimize the meeting's progress. This means the server can acquire emotional data in real time and appropriately adjust the meeting's progress, for example, by prompting participants to take a break if their concentration is waning.

[1538] Finally, the server has a means of managing customer service schedules and sending reminder notifications. This allows the server to manage customer service appointments and automatically send reminder notifications, enabling staff to efficiently handle multiple customers.

[1539] As a specific example, when a staff member wearing smart glasses efficiently attends to multiple customers, the following prompt message will be displayed.

[1540] "Current customers want to learn more about the products they've shown interest in. Provide them with relevant information and, if necessary, offer customized suggestions."

[1541] This system allows businesses to understand customers' emotional states in real time at physical stores, enabling them to respond quickly and optimally, thereby improving the quality of customer service.

[1542] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1543] Step 1:

[1544] Input: The user submits a request for an online meeting.

[1545] Action: The server receives this request and saves its contents to the database. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1546] Output: Request information is saved in the database.

[1547] Step 2:

[1548] Input: The server accesses each user's calendar API to retrieve the meeting participants' schedule information.

[1549] Operation: The server retrieves the schedule information of each participant and uses that information to calculate the date and time when the largest number of participants are available.

[1550] Output: The server notifies the user of the list of optimal date and time options.

[1551] Step 3:

[1552] Input: The server notifies the user of possible dates and times, and the user confirms the final date and time.

[1553] Action: The date and time confirmed by the user will be finalized as the meeting date and time.

[1554] Output: The final date and time are confirmed and notified to meeting participants.

[1555] Step 4:

[1556] Input: The server accesses shared folders and external databases to collect materials and data related to the meeting agenda.

[1557] Operation: The server organizes the collected data and uploads it to a shared folder in a format accessible to all participants.

[1558] Output: Necessary documents and data are saved in a shared folder.

[1559] Step 5:

[1560] Input: At the start of the meeting, the server sends a reminder notification to all participants.

[1561] Function: Sends reminder notifications to multiple devices to encourage meeting preparations.

[1562] Output: A reminder notification will be sent to all participants before the meeting starts.

[1563] Step 6:

[1564] Input: The server manages the scheduled time for each agenda item during the meeting and presents relevant data in real time.

[1565] Function: Manages time according to the progress of the meeting and displays necessary information in real time.

[1566] Output: Participants can immediately access the data they need.

[1567] Step 7:

[1568] Input: The server records the audio during the meeting.

[1569] Operation: The server uses speech recognition technology to automatically convert recorded audio data into text.

[1570] Output: The transcribed meeting content is generated as meeting minutes.

[1571] Step 8:

[1572] Input: The server automatically generates meeting minutes based on the generated text.

[1573] Operation: The server automatically generates meeting minutes and organizes them according to a pre-distributed template.

[1574] Output: Meeting minutes will be distributed to all participants via email and in-system notifications.

[1575] Step 9:

[1576] Input: The server uses an emotion analysis engine to recognize the emotional state of participants in real time.

[1577] Operation: The server uses sentiment data to optimize the meeting's progress and adjust the flow as needed. For example, it might prompt participants to take a break if their concentration is waning.

[1578] Output: Optimized meeting management is achieved.

[1579] Step 10:

[1580] Input: The server automatically extracts the implementation plan from the meeting minutes.

[1581] Operation: The server analyzes the implementation plan and assigns it to the responsible person.

[1582] Output: The implementation plan is assigned to the person in charge.

[1583] Step 11:

[1584] Input: The server retrieves data for managing customer service schedules.

[1585] Operation: The server manages customer service schedules and automatically sends reminder notifications.

[1586] Output: A reminder notification will be sent at the specified time.

[1587] The specific software used in each step includes MySQL for database management, the Google Calendar API for communication with the calendar API, the Google Cloud Speech-to-Text API for speech recognition, DeepFace for sentiment analysis, and Python program code for controlling the system.

[1588] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1589] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1590] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1591] [Fourth Embodiment]

[1592] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1593] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1594] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1595] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1596] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1597] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1598] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1599] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1600] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1601] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1602] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1603] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1604] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1605] The system of this invention combines multiple functions to enable the efficient and automated operation of online meetings. This system is centered around a server, terminals, and users, each playing a specific role.

[1606] 1. Accepting requests

[1607] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1608] The server receives this request and saves it to the database. This data is used for subsequent processing.

[1609] 2. Scheduling

[1610] The server communicates with each user's calendar API to retrieve the calendar information of all participants.

[1611] Based on the acquired calendar information, the server calculates the optimal date and time. For example, it lists dates and times when many participants are available.

[1612] These proposed dates and times will be notified to the users for each participant to review. The date and time will be finalized upon final approval.

[1613] 3. Meeting preparations

[1614] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[1615] The server organizes the collected data and uploads it to a shared folder.

[1616] Furthermore, the server creates meeting minutes templates based on the agenda items and distributes them to all participants.

[1617] 4. Meeting Management

[1618] The server sends a reminder notification to the user at the start of the meeting.

[1619] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data to the user's device in real time, depending on the agenda item.

[1620] 5. Recording of discussions and creation of meeting minutes.

[1621] The device records the meeting content and sends the data to the server.

[1622] The server converts the audio data into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[1623] 6. Action plan generation and management

[1624] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the responsible person.

[1625] Users manage the progress of their assigned action plans and update them through the system as needed.

[1626] The server will periodically report the progress of the action plan to all participants.

[1627] Specific example

[1628] For example, suppose a company wants to hold an online meeting for a new product development project. In this case, the user proposes the meeting regarding the new product development, enters the date and time, participant list, and agenda into the system, and submits the request.

[1629] The server receives the request and retrieves each participant's calendar information. It calculates the optimal date and time and notifies everyone. All participants review the proposed dates and times and finalize the date and time.

[1630] Before the meeting, terminals collect market research data and competitor analysis materials, and the server organizes and uploads them to a shared folder. Additionally, meeting minute templates tailored to the agenda are generated and distributed to participants in advance.

[1631] During the meeting, the server keeps track of the time and displays relevant data in real time for each agenda item. The meeting content is recorded by the terminals, and the server converts the audio data into text to automatically create meeting minutes.

[1632] After the meeting ends, the server extracts action plans from the meeting minutes and assigns them to the responsible parties. Users manage and update the progress of the action plans within the system, and the server reports the progress to all participants.

[1633] As a result, it becomes possible to streamline and automate the entire process, from meeting preparation to conducting and follow-up.

[1634] The following describes the processing flow.

[1635] Step 1:

[1636] A user submits an online meeting request to the system. The request includes details such as the date and time of the meeting, a list of participants, and the agenda.

[1637] Step 2:

[1638] The server receives this request and saves it to the database. This data is used for subsequent processing.

[1639] Step 3:

[1640] The server communicates with each user's calendar API to retrieve calendar information for all participants. This calendar information includes each user's schedule and availability.

[1641] Step 4:

[1642] The server calculates the optimal date and time based on the calendar information it obtains. This date and time is listed as the time when the largest number of participants are available.

[1643] Step 5:

[1644] The server notifies users of a list of possible dates and times, and each participant confirms them. Based on the participants' confirmations, the optimal date and time are finalized.

[1645] Step 6:

[1646] The terminal collects materials and data related to the meeting agenda. This includes retrieving information from shared folders and collecting it from external databases.

[1647] Step 7:

[1648] The server organizes the collected data and uploads it to a shared folder. The data is shared in a format accessible to all participants.

[1649] Step 8:

[1650] The server generates a meeting minutes template and distributes it to all participants in advance. The minutes template includes slots for each agenda item.

[1651] Step 9:

[1652] The server sends a reminder notification to the user at the start of the meeting. This notification is sent to multiple devices.

[1653] Step 10:

[1654] During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It displays the remaining time in real time according to the progress of each agenda item.

[1655] Step 11:

[1656] The server presents relevant data in real time according to the agenda. The presented data is displayed in a format that participants can immediately refer to.

[1657] Step 12:

[1658] The device records the meeting content and sends the data to the server. The recorded data is saved as an audio file.

[1659] Step 13:

[1660] The server converts the recorded audio data into text. Text is automatically generated using speech recognition technology.

[1661] Step 14:

[1662] The server automatically generates meeting minutes based on the generated text. The minutes are organized according to a template distributed in advance.

[1663] Step 15:

[1664] After the meeting ends, the server distributes the generated minutes to all participants. The minutes are sent to each participant via email or in-system notification.

[1665] Step 16:

[1666] The server analyzes the meeting minutes to create an action plan and automatically assigns it to the appropriate person. The action plan includes specific tasks and assignees.

[1667] Step 17:

[1668] Manage the progress of action plans assigned to users. Update task progress on the system.

[1669] Step 18:

[1670] The server reports the progress of the action plan to all participants. Regular notifications are sent to share progress.

[1671] The above is the specific processing flow of the program for this system.

[1672] (Example 1)

[1673] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1674] Current online meeting systems make it difficult to efficiently manage the entire process, from meeting preparation to execution and follow-up. Furthermore, tasks such as timekeeping, real-time presentation of relevant materials during meetings, and post-meeting minute-taking and action plan extraction and management are not automated, relying heavily on manual processes. This results in significant effort and time being required for effective meeting management.

[1675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1676] In this invention, the server includes means for receiving meeting requests, means for obtaining the schedule information of meeting participants and calculating the optimal date and time, means for collecting and sharing materials and information related to the meeting, means for managing the scheduled time for each agenda item during the meeting and presenting relevant information in real time, means for recording the meeting content, automatically transcribing it into text to generate meeting minutes, means for automatically extracting action plans from the meeting minutes and assigning them to responsible persons, means for sending reminders at the start of the meeting, means for sending meeting recording data to the server, and means for periodically reporting the progress of the action plans. This makes it possible to streamline and automate the entire process from meeting preparation to execution and follow-up.

[1677] "Means for receiving meeting requests" refers to the means by which users propose online meetings and send detailed information such as the date and time, participant list, and agenda to the server.

[1678] "A means of obtaining the schedule information of meeting participants and calculating the optimal date and time" refers to a method in which the server communicates with each participant's calendar API and calculates the optimal date and time that all participants can attend based on the information obtained.

[1679] "Means for collecting and sharing materials and information related to a meeting" refers to a method by which terminals collect necessary materials for a meeting from shared folders or external databases, and a server organizes and shares them with participants.

[1680] "A means of managing scheduled time for each agenda item during a meeting and presenting relevant information in real time" refers to a method in which a server manages the scheduled time during the meeting and presents relevant materials and information in real time as needed.

[1681] "A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes" refers to a method in which a terminal records the audio during a meeting, and a server uses speech recognition technology to transcribe that data into text and compile it into meeting minutes.

[1682] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a method in which a server analyzes the text of the meeting minutes, extracts specific action items, and automatically assigns them to the appropriate personnel.

[1683] "Method for sending reminders at the start of a meeting" refers to a method by which the server sends a reminder to all participants to notify them of the start time as the meeting is about to begin.

[1684] "Means for sending meeting recording data to the server" refers to the means by which a terminal uploads the recorded meeting audio data to the server.

[1685] "Means for periodically reporting on the progress of the action plan" refers to a method by which the server periodically checks the implementation status of the extracted action plan and reports its progress to all participants.

[1686] The system of the present invention enables the efficient and automated operation of online meetings through the collaborative roles of the server, terminal, and user. Specific embodiments are described below.

[1687] Accepting requests

[1688] A user enters and submits an online meeting request. This includes details such as the date and time, participant list, and agenda. For example, suppose a user submits a request for "Meeting on October 15, 2023 at 2:00 PM, with participants A, B, and C, on the topic of new product development." The server receives this request and stores it in the database. This makes the meeting details available for subsequent processing.

[1689] Example prompt:

[1690] I would like to hold a meeting regarding new product development on October 15, 2023, at 2:00 PM. The participants are Person A, Person B, and Person C. The topic of discussion will be "New Product Development".

[1691] Schedule adjustment

[1692] The server accesses the calendar APIs of all participants (e.g., Google Calendar API, Outlook Calendar API) to retrieve calendar information. Based on this information, the server calculates the optimal date and time and notifies the users of the suggested dates and times. Each user who receives the notification reviews the suggested dates and times and accepts the optimal one. Through this confirmation, the final date and time are determined.

[1693] Example prompt:

[1694] Please obtain each participant's calendar information, calculate a date and time that works for everyone, and notify them accordingly.

[1695] Meeting preparations

[1696] The terminal collects materials and data related to the meeting. These are collected from shared folders (e.g., Google Drive, Dropbox) or external databases (e.g., a company's internal database). The server organizes these materials and uploads them to the shared folder, as well as generating meeting minutes templates based on the agenda and distributing them to participants in advance.

[1697] Example prompt:

[1698] Collect market research data and competitive analysis materials, upload them to a shared folder, and generate a meeting minutes template to distribute to participants.

[1699] Meeting progress

[1700] The server sends a reminder notification at the start of the meeting. During the meeting, the server manages the scheduled time for each agenda item and keeps track of the time. It also displays relevant data related to the agenda item on the user's device in real time. For example, during a discussion on new product development, relevant market research data is automatically displayed.

[1701] Example prompt:

[1702] Send reminder notifications at the start of the meeting, manage the scheduled time for each agenda item, and present relevant data in real time.

[1703] Recording of discussions and creation of meeting minutes

[1704] The terminal records the meeting content and sends the data to the server. The server uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the recording into text and automatically generates meeting minutes. The generated minutes are organized and distributed to all participants after the meeting.

[1705] Example prompt:

[1706] Please record the meeting, transcribe the audio data into text, and automatically create and distribute meeting minutes.

[1707] Action plan generation and management

[1708] The server extracts action plans from meeting minutes and automatically assigns them to the appropriate personnel. Natural language processing technology is used to identify important tasks and assign them to the correct individuals. Each user manages and updates the progress of their assigned action plan within the system. The server periodically reports the progress to all participants.

[1709] Example prompt:

[1710] Extract action plans from the meeting minutes, assign them to the responsible parties, and have them report on the progress regularly.

[1711] As described above, the system of the present invention can streamline and automate the entire process from preparation to implementation and follow-up of online meetings. Through this system, the effort and time required for meeting management can be significantly reduced.

[1712] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1713] Step 1:

[1714] The user enters and submits an online meeting request. The user fills in details such as the date and time, participant list, and agenda in the form and presses the "Submit" button. The input data includes the meeting date and time, participant list, and agenda.

[1715] Specific actions:

[1716] The user opens a web form.

[1717] Fill in the details of the meeting in the input field.

[1718] Click the "Send" button.

[1719] Output: Meeting request data received by the server.

[1720] Step 2:

[1721] The server receives requests from users and stores them in the database. The server receives HTTP POST requests, parses their contents, and adds new records to the database.

[1722] Specific actions:

[1723] The server waits for HTTP requests.

[1724] User request received.

[1725] The request content is analyzed and saved to the "Meeting Requests" table.

[1726] Input: Meeting request data from the user.

[1727] Output: Meeting request information stored in the database.

[1728] Step 3:

[1729] The server communicates with all participants' calendar APIs to retrieve calendar information. The server uses each participant's authentication token to access the calendar API and retrieve event information.

[1730] Specific actions:

[1731] The server sends a request to the calendar API.

[1732] Retrieve the calendar information of each participant.

[1733] Analyze the calendar information to list available dates and times.

[1734] Input: Participant authentication token.

[1735] Output: Participant calendar information.

[1736] Step 4:

[1737] Based on the calendar information obtained by the server, the optimal date and time for all participants to attend is calculated. These suggested dates and times are then notified to the users.

[1738] Specific actions:

[1739] Analyze the acquired calendar information.

[1740] The algorithm calculates the dates and times when everyone is free.

[1741] The notification sending function will send the user a list of possible dates and times.

[1742] Input: Participant's calendar information.

[1743] Output: A list of possible dates and times sent to the user.

[1744] Step 5:

[1745] The user reviews the suggested dates and times and approves the most suitable one. Upon receiving the notification, the user selects the appropriate date and time from the list of options and replies to the system.

[1746] Specific actions:

[1747] The user opens the notification.

[1748] Check the suggested dates and times.

[1749] Select a date and time for approval and reply to the system.

[1750] Input: A list of possible dates and times from the server.

[1751] Output: Final date and time of approval.

[1752] Step 6:

[1753] The device collects meeting-related materials and data. It accesses shared folders (e.g., Google Drive, Dropbox) and external databases to retrieve and organize necessary materials.

[1754] Specific actions:

[1755] The device accesses shared folders and external databases.

[1756] Download the necessary documents.

[1757] Classify and organize the documents.

[1758] Input: Access information for data collection.

[1759] Output: Collected and organized materials.

[1760] Step 7:

[1761] The server uploads the collected materials to a shared folder, creates a meeting minutes template based on the agenda, and distributes it to the participants.

[1762] Specific actions:

[1763] The server uploads the documents to the shared folder.

[1764] Create a meeting minutes template using the template generation module.

[1765] Distribute the generated template to participants via email or shared link.

[1766] Input: Collected and organized materials.

[1767] Output: Documents uploaded to the shared folder, meeting minutes templates distributed to participants.

[1768] Step 8:

[1769] The server will send a reminder notification when the meeting starts.

[1770] Specific actions:

[1771] The server executes code that sends a reminder 5 minutes before the start time.

[1772] A reminder notification will be sent to each participant's device.

[1773] Input: Meeting start time information.

[1774] Output: Reminder notifications sent to participants.

[1775] Step 9:

[1776] The server manages the scheduled meeting time and keeps track of it. During the meeting, it displays relevant data to the user's terminal in real time, depending on the agenda.

[1777] Specific actions:

[1778] The server sets a timer and manages the scheduled time.

[1779] Timekeeping alerts are sent to each user's device.

[1780] Relevant data will be delivered in real time.

[1781] Input: Scheduled time information for each agenda item.

[1782] Output: Real-time data displayed on the user's device.

[1783] Step 10:

[1784] The device records the meeting content and sends the data to the server.

[1785] Specific actions:

[1786] The device launches the voice recording app.

[1787] Record the meeting content and save it to local storage.

[1788] Upload the recording file to the server.

[1789] Input: Audio data from a meeting.

[1790] Output: Recording file uploaded to the server.

[1791] Step 11:

[1792] The server uses a speech recognition API to convert the audio data into text and generate meeting minutes.

[1793] Specific actions:

[1794] The server sends audio data to the speech recognition API.

[1795] The acquired text data is formatted and compiled into meeting minutes.

[1796] Input: Audio file.

[1797] Output: Generated meeting minutes.

[1798] Step 12:

[1799] The server extracts action plans from meeting minutes and automatically assigns them to responsible personnel. The server uses natural language processing technology to identify important tasks and notifies the responsible personnel.

[1800] Specific actions:

[1801] The server analyzes the meeting minutes and extracts the tasks.

[1802] Assign the extracted tasks to the responsible personnel.

[1803] Input: Generated meeting minutes.

[1804] Output: Action plan assigned to the person in charge.

[1805] Step 13:

[1806] Users manage the progress of their assigned action plans and update them through the system as needed.

[1807] Specific actions:

[1808] The user logs into the system.

[1809] Use the interface to update the progress of the action plan.

[1810] Update the progress status.

[1811] Input: Progress status of the action plan.

[1812] Output: Updated progress.

[1813] Step 14:

[1814] The server will periodically report the progress of the action plan to all participants.

[1815] Specific actions:

[1816] The server checks the progress at regular intervals.

[1817] Summarize the progress and generate a report.

[1818] Send the report to all participants.

[1819] Input: Updated progress status.

[1820] Output: Progress report sent to all participants.

[1821] (Application Example 1)

[1822] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1823] Traditional online meeting systems faced challenges such as requiring significant time and effort for meeting preparation, execution, and follow-up. Furthermore, recording meetings and creating minutes were cumbersome, making efficient operation difficult. This was especially true when multiple participants were involved, as smoothly managing all processes—scheduling, organizing materials, automatically generating minutes, and managing action plans—became even more challenging. As a result, the quality of meetings often suffered, hindering the maximization of results.

[1824] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1825] In this invention, the server includes means for receiving meeting requests, means for obtaining calendar information of meeting participants and calculating the optimal date and time, means for collecting meeting-related materials and data, organizing them in a shared folder and uploading them, means for managing scheduled times for each agenda item and presenting relevant data to the user terminal in real time, means for recording meeting content, automatically transcribing it into text and generating meeting minutes, means for automatically extracting action plans from meeting minutes and assigning them to responsible persons, means for communicating with participants' calendars to calculate the optimal date and time and notifying the user, means for collecting and organizing relevant data and sharing a webinar page automatically created using a template, means for providing relevant information in real time during the meeting, and means for inputting prompt sentences using a generation AI model and transcribing audio data into text. This makes it possible to streamline and automate the entire process from meeting preparation to progress and follow-up.

[1826] "Means of receiving meeting requests" refers to a function where a user proposes an online meeting, sends detailed information such as the date and time, participant list, and agenda to the system, and the server receives the request.

[1827] The "method for obtaining meeting participants' calendar information and calculating the optimal date and time" is a function that communicates with each meeting participant's calendar API, checks everyone's availability, and calculates the most suitable date and time for the meeting.

[1828] "A means of collecting meeting-related materials and data, organizing them in a shared folder, and uploading them" refers to a function that collects necessary materials and data from different sources, organizes them, and uploads them to a single shared folder.

[1829] "A means of managing scheduled time for each agenda item and presenting relevant data to the user's terminal in real time" refers to a function that manages the scheduled time allocated to each agenda item and displays relevant data on the user's terminal in real time during the meeting.

[1830] "A method for recording meeting content, automatically converting it to text, and generating meeting minutes" refers to a function that records audio during a meeting, automatically converts that audio data into text, and generates meeting minutes.

[1831] "A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel" refers to a function that extracts specific action plans from generated meeting minutes and automatically assigns them to the responsible personnel.

[1832] "A means of communicating with participants' calendars to calculate the optimal date and time and notifying the user" refers to a function that communicates with meeting participants' calendar information, calculates multiple date and time options, and notifies the user.

[1833] "A means of collecting and organizing relevant data and sharing a webinar page automatically created using a template" refers to a function that collects the necessary data related to a webinar, organizes it, and then automatically generates and shares a webinar page using a template.

[1834] "Means of providing relevant information in real time during a meeting" refers to a function that provides information and materials related to the agenda in real time while the meeting is in progress.

[1835] "A means of inputting prompt sentences using a generative AI model and converting audio data to text" refers to a function that utilizes a generative AI model to input prompt sentences for converting audio data to text, and then converts the audio data to text based on these prompts.

[1836] The present invention provides a multi-functional system that enables the efficient and automated operation of online meetings. In this system, the server, terminals, and users each play specific roles and work together.

[1837] System Configuration

[1838] hardware

[1839] Smartphones (iOS / Android): Devices used by users and meeting participants.

[1840] Server: A central computer that processes and manages data.

[1841] Internet connection: Responsible for communication between all devices.

[1842] software

[1843] Client-side: React Native

[1844] Server-side: Node.js, Express

[1845] Database: MongoDB

[1846] AI speech recognition: Google Cloud Speech-to-Text API

[1847] Calendar APIs: Google Calendar API, Microsoft Outlook API

[1848] Function flow

[1849] 1. Means of receiving meeting requests

[1850] A user sends an online meeting request to the server using a smartphone application. This request includes detailed information such as the date and time of the meeting, a list of participants, and the agenda. The server receives this information and stores it in its database.

[1851] 2. A means of obtaining the calendar information of meeting participants and calculating the optimal date and time.

[1852] The server communicates with each meeting participant's calendar API (such as Google Calendar API or Microsoft Outlook API) to retrieve everyone's availability. Based on the retrieved calendar information, the server calculates the optimal date and time and notifies the user.

[1853] 3. A method for collecting meeting-related materials and data, organizing them in a shared folder, and uploading them.

[1854] The user's device collects the necessary materials and data for the meeting, and the server organizes them and uploads them to a shared folder. This shared folder is accessible to the user and meeting participants.

[1855] 4. A means of managing the scheduled time for each agenda item and presenting relevant data to the user's terminal in real time.

[1856] The server manages the scheduled time for each agenda item during the meeting and tracks the progress of time. It also displays relevant data to the user's smartphone in real time.

[1857] 5. A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes.

[1858] The user's smartphone records the meeting and sends the audio data to the server. The server uses the Google Cloud Speech-to-Text API to convert the audio data into text and automatically generates meeting minutes.

[1859] 6. A method for automatically extracting action plans from meeting minutes and assigning them to responsible personnel.

[1860] The server extracts action plans from the generated meeting minutes and assigns them to responsible parties. Users then manage the progress of these action plans based on these plans.

[1861] 7. Means of providing relevant information in real time

[1862] The server provides users with relevant information and materials in real time during the meeting, based on the agenda. This feature ensures that the meeting proceeds smoothly.

[1863] 8. A method for inputting prompt text using a generative AI model and converting speech data into text.

[1864] Users can leverage a generative AI model to input appropriate prompts and efficiently convert audio data into text.

[1865] Specific example

[1866] When a company holds an online webinar to announce a new product, this system automates and streamlines all preparation and execution. For example, a user submits a request for a new product announcement webinar to the system, and the server retrieves each participant's calendar information to calculate the optimal date and time. Relevant materials are collected, and the webinar page is automatically generated based on an organized template. During the meeting, relevant information is provided in real time, and after the meeting, the audio data is transcribed into text to generate meeting minutes.

[1867] Examples of prompts to input into a generative AI model:

[1868] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new product features, market strategy, and Q&A after the session. The audio recording is available below."

[1869] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1870] Step 1:

[1871] A user sends an online meeting request via a smartphone app. The user enters detailed information such as the meeting date and time, participant list, and agenda. This information is then sent to a server and stored in a database. The database contains fields for meeting ID, date and time, participant list, and agenda. Specifically, the user fills in the required information in the app's input form and presses the submit button.

[1872] Step 2:

[1873] The server retrieves participants' calendar information based on meeting information stored in the database. As input, the server sends requests to each participant's calendar API (such as Google Calendar API or Microsoft Outlook API) based on the meeting ID and participant list. As output, the server receives each participant's free time. Specifically, the server retrieves each participant's calendar information in parallel and compiles everyone's free time into a list.

[1874] Step 3:

[1875] The server calculates the optimal meeting date and time based on the retrieved calendar information. It receives a list of participants' available time slots as input. As output, it generates a list of the optimal dates and times when all participants are available. Specifically, the server compares the available time slot lists and executes logic to select the date and time when the most participants are available.

[1876] Step 4:

[1877] The server notifies the user of the calculated optimal date and time. As input, it generates a notification message based on a list of optimal dates and times. As output, this notification message is sent to the user's smartphone, making it available for the user to view. Specifically, the server sends the message to the smartphone via a push notification API.

[1878] Step 5:

[1879] The terminal collects the necessary materials and data for the meeting and sends them to the server. As input, it receives access links to materials provided by users and meeting participants, as well as uploaded files. As output, the materials and data are uploaded to the server and organized in a shared folder. Specifically, users use the app's material upload function to select and submit the necessary files.

[1880] Step 6:

[1881] The server organizes and uploads received documents and data to a shared folder. It receives collected documents and data as input. As output, these are categorized and saved in a designated folder, making them accessible to participants. Specifically, the server automatically generates a directory structure and appropriately places the documents.

[1882] Step 7:

[1883] The server manages the scheduled time and agenda during a meeting. It receives the start time and the scheduled time for each agenda item as input. Its output includes real-time progress tracking and displays warnings as the scheduled end time approaches. Specifically, the server uses a timer function to monitor time and issues alerts each time the scheduled time is reached.

[1884] Step 8:

[1885] The server records the meeting content and sends the audio data to the Google Cloud Speech-to-Text API for transcription. It receives the recorded audio data as input and automatically generated text data as output. Specifically, the server sends the audio file to the API and converts the returned text data into meeting minutes.

[1886] Step 9:

[1887] The server extracts action plans from meeting minutes and assigns them to responsible personnel. It analyzes the transcribed meeting minutes as input. The output determines each action item and its assigned person. Specifically, the server uses natural language processing to analyze the meeting minutes and automatically generates and assigns action plans.

[1888] Step 10:

[1889] The server uses a generative AI model to take prompt text as input and provides a means to convert audio data into text. As input, audio data is sent to the model along with the prompt text. As output, text data based on the prompt is generated. Specifically, the server appropriately constructs the prompt text and performs immediate analysis on the AI ​​model.

[1890] Examples of prompts to input into a generative AI model:

[1891] "Please create minutes for the new product launch webinar. The date was October 10, 2023, the participants are listed below, and the agenda includes new pr...

Claims

1. In online meetings, the means of receiving meeting requests, A method for obtaining the calendar information of meeting participants and calculating the optimal date and time, A means of collecting and sharing materials and data related to the meeting, A method for managing the scheduled time for each agenda item during a meeting and presenting relevant data in real time, A method for recording meeting content, automatically transcribing it into text, and generating meeting minutes, A method for automatically extracting action plans from meeting minutes and assigning them to the responsible person, A system that includes this.

2. The system according to claim 1, further comprising means for automatically sending reminder notifications to meeting participants.

3. The system according to claim 1, further comprising means for managing and reporting on the progress of an action plan.

Images